đ đ˛đŗ Mongol mGPT 1.3B
Mongol mGPT 1.3B is a language model specifically designed for the Mongol language. As the name suggests, this model has 1.3 billion parameters.
The Mongol language belongs to the Mongolic language family. It's a language with a long - standing history and is spoken by approximately 5.7 million people. Here are some key facts about it:
- It is the official language of Mongolia.
- In Mongolia, it uses the Cyrillic script, while the traditional Mongolian script is still in use in other regions.
- It has a rich history closely associated with the Mongol Empire and historical figures like Genghis Khan.
đ Quick Start
The provided README doesn't have specific quick - start steps. If you want to use this model, you can refer to the Hugging Face documentation related to mGPT models for general guidance on model loading and inference.
⨠Features
- Specifically tailored for the Mongol language, enabling more accurate language processing for Mongol - related tasks.
- Derived from a well - trained base model, which benefits from pre - training on a diverse set of languages.
đĻ Installation
The original README doesn't contain installation steps. You may follow the general installation process for Hugging Face models. Usually, you can use the transformers library in Python:
pip install transformers
đ§ Technical Details
This model is one of the derivatives of the base [mGPT - XL (1.3B)](https://huggingface.co/ai - forever/mGPT) model. The base model was initially trained on 61 languages from 25 language families, using Wikipedia and the C4 corpus.
We discovered additional data for 23 languages, most of which are considered minor languages. Then, we decided to further fine - tune the base model. Mongol mGPT 1.3B was trained for an additional 50,000 steps with a batch size of 4 and a context window of 2048 tokens on 1 A100 GPU.
The final perplexity of this model on the validation set is 4.35.
Chart of the training loss and perplexity:
đ Documentation
Other mGPT - 1.3B models
- [đĻđ˛ mGPT - 1.3B Armenian](https://huggingface.co/ai - forever/mGPT - 1.3B - armenian)
- [đĻđŋ mGPT - 1.3B Azerbaijan](https://huggingface.co/ai - forever/mGPT - 1.3B - azerbaijan)
- [đ¯ mGPT - 1.3B Bashkir](https://huggingface.co/ai - forever/mGPT - 1.3B - bashkir)
- [đ§đž mGPT - 1.3B Belorussian](https://huggingface.co/ai - forever/mGPT - 1.3B - belorussian)
- [đ§đŦ mGPT - 1.3B Bulgarian](https://huggingface.co/ai - forever/mGPT - 1.3B - bulgarian)
- [đ mGPT - 1.3B Buryat](https://huggingface.co/ai - forever/mGPT - 1.3B - buryat)
- [đŗ mGPT - 1.3B Chuvash](https://huggingface.co/ai - forever/mGPT - 1.3B - chuvash)
- [đŦđĒ mGPT - 1.3B Georgian](https://huggingface.co/ai - forever/mGPT - 1.3B - georgian)
- [đ¸ mGPT - 1.3B Kalmyk](https://huggingface.co/ai - forever/mGPT - 1.3B - kalmyk)
- [đ°đŋ mGPT - 1.3B Kazakh](https://huggingface.co/ai - forever/mGPT - 1.3B - kazakh)
- [đ°đŦ mGPT - 1.3B Kirgiz](https://huggingface.co/ai - forever/mGPT - 1.3B - kirgiz)
- [đģ mGPT - 1.3B Mari](https://huggingface.co/ai - forever/mGPT - 1.3B - mari)
- [đ mGPT - 1.3B Ossetian](https://huggingface.co/ai - forever/mGPT - 1.3B - ossetian)
- [đŽđˇ mGPT - 1.3B Persian](https://huggingface.co/ai - forever/mGPT - 1.3B - persian)
- [đˇđ´ mGPT - 1.3B Romanian](https://huggingface.co/ai - forever/mGPT - 1.3B - romanian)
- [đšđ¯ mGPT - 1.3B Tajik](https://huggingface.co/ai - forever/mGPT - 1.3B - tajik)
- [â mGPT - 1.3B Tatar](https://huggingface.co/ai - forever/mGPT - 1.3B - tatar)
- [đšđ˛ mGPT - 1.3B Turkmen](https://huggingface.co/ai - forever/mGPT - 1.3B - turkmen)
- [đ mGPT - 1.3B Tuvan](https://huggingface.co/ai - forever/mGPT - 1.3B - tuvan)
- [đēđĻ mGPT - 1.3B Ukranian](https://huggingface.co/ai - forever/mGPT - 1.3B - ukranian)
- [đēđŋ mGPT - 1.3B Uzbek](https://huggingface.co/ai - forever/mGPT - 1.3B - uzbek)
- [đ mGPT - 1.3B Yakut](https://huggingface.co/ai - forever/mGPT - 1.3B - yakut)
đ License
This model is released under the MIT license.
Feedback
If you find a bug or have additional data for training the model on your language, please provide us with feedback.
The model will be improved over time. Stay tuned!
đ Information Table
| Property |
Details |
| Model Type |
Mongol mGPT 1.3B |
| Training Data |
Additional data for 23 languages, base model trained on Wikipedia and C4 corpus for 61 languages from 25 language families |
| License |
MIT |
%20--%3e%3cdefs%3e%3cstyle%3e%20.st0%20{%20fill:%20%23061b40;%20}%20.st1%20{%20fill:%20%23306af1;%20}%20.st2%20{%20fill:%20%235ce5cf;%20}%20%3c/style%3e%3c/defs%3e%3cg%3e%3cpath%20class='st0'%20d='M55,10.5h9v3h-9v9h12V7.5h-12v3ZM64,19.5h-6v-3h6v3Z'/%3e%3cpolygon%20class='st0'%20points='69%2016.5%2078%2016.5%2078%2019.5%2069%2019.5%2069%2022.5%2081%2022.5%2081%2013.5%2072%2013.5%2072%2010.5%2081%2010.5%2081%207.5%2069%207.5%2069%2016.5'/%3e%3cpolygon%20class='st0'%20points='95%2010.5%2095%207.5%2083%207.5%2083%2022.5%2095%2022.5%2095%2019.5%2086%2019.5%2086%2016.5%2095%2016.5%2095%2013.5%2086%2013.5%2086%2010.5%2095%2010.5'/%3e%3cpath%20class='st0'%20d='M40,1.5v21h11.6l1.4-1.4v-7.6h0c0,0-1.4-1.5-1.4-1.5l1.4-1.4V2.9l-1.4-1.4h-11.6ZM50,19.5h-7v-6h7v6ZM50,10.5h-7v-6h7v6Z'/%3e%3c/g%3e%3cpath%20class='st1'%20d='M23.1,24L14.7,4.8l-4.9,11.2h3.8l-1.8,4H3.3L12.1,0H2C.9,0,0,.9,0,2v20c0,1.1.9,2,2,2h21.1Z'/%3e%3cpath%20class='st2'%20d='M34,0h-16.8l10.6,24h6.2c1.1,0,2-.9,2-2V2C36,.9,35.1,0,34,0ZM32.5,20h-4V4h4v16Z'/%3e%3c/svg%3e)
