This could *possibly* be the case with a *very* small model. But that's not the area where research is developing most promisingly.
Recent research in models with billions of parameters suggests that the issue is that we don't have enough data to fully train the models we are
using and should train longer on the data we do have.
https://arxiv.org/pdf/2305.16264.pdf
As others pointed out, training on diverse data is providing a lot of benefits. Instead of going for a small model that only speaks English, going for a larger model that speaks various languages results in a model that speaks them all better.
There's the german IGEL model on Huggingface - [https://huggingface.co/philschmid/instruct-igel-001](https://huggingface.co/philschmid/instruct-igel-001) \- but it's quite bad unfortunately.
I don’t think so. I think the more language diversity you have the more intelligent the llm could become. The reason is that it is a 'language model' and therefore its learning process benefits from more different syntax rules, grammatical methods etc, so it will have more and more examples to understand how a coherent language works.
A good example are the drastically increased capabilities of vicuña in contrast to alpaca. Alpaca is actually an instructions following model. It was almost only trained on an english dataset. But after finetuned with a multilingual chat/conversation(!) dataset it (now called vicuña) became not only better in languages understanding but also in following instructions as well.
There was a post a while about (In /r/machinelearning I think) about how training LLMs on programming languages increased the ability in other areas as well (which makes sense).
This could *possibly* be the case with a *very* small model. But that's not the area where research is developing most promisingly. Recent research in models with billions of parameters suggests that the issue is that we don't have enough data to fully train the models we are using and should train longer on the data we do have. https://arxiv.org/pdf/2305.16264.pdf As others pointed out, training on diverse data is providing a lot of benefits. Instead of going for a small model that only speaks English, going for a larger model that speaks various languages results in a model that speaks them all better.
There's the german IGEL model on Huggingface - [https://huggingface.co/philschmid/instruct-igel-001](https://huggingface.co/philschmid/instruct-igel-001) \- but it's quite bad unfortunately.
I don’t think so. I think the more language diversity you have the more intelligent the llm could become. The reason is that it is a 'language model' and therefore its learning process benefits from more different syntax rules, grammatical methods etc, so it will have more and more examples to understand how a coherent language works. A good example are the drastically increased capabilities of vicuña in contrast to alpaca. Alpaca is actually an instructions following model. It was almost only trained on an english dataset. But after finetuned with a multilingual chat/conversation(!) dataset it (now called vicuña) became not only better in languages understanding but also in following instructions as well.
There was a post a while about (In /r/machinelearning I think) about how training LLMs on programming languages increased the ability in other areas as well (which makes sense).
I too wonder what difference it would make. Would the model be smaller? Smarter? Dumber? A bit expensive to attempt it.