Mgpt 1.3B Kazakh

🚀 🇰🇿 Kazakh mGPT 1.3B

This is a language model designed for the Kazakh language. As the name suggests, the model has 1.3 billion parameters.

Kazakh belongs to the Turkic language family. It is a dynamic language spoken by approximately 18 million people. Here are some key facts about it:

1. It is a major language in Kazakhstan.
2. Kazakh uses a version of the Cyrillic script but is in the process of transitioning to the Latin script.
3. It has a rich heritage of oral literature, including epic poetry.

✨ Features

• Specifically tailored for the Kazakh language.
• Derived from a well - trained base model with further fine - tuning.

📚 Documentation

🔧 Technical Details

This model is one of the derivatives of the base [mGPT - XL (1.3B)](https://huggingface.co/ai - forever/mGPT) model (see the list below). The base model was initially trained on 61 languages from 25 language families using the Wikipedia and C4 corpus.

We discovered additional data for 23 languages, most of which are considered minor languages, and decided to further fine - tune the base model. Kazakh mGPT 1.3B was trained for an additional 150,000 steps with a batch_size of 4 and a context window of 2048 tokens on 1 A100.

The final perplexity of this model on the validation set is 3.38.

Chart of the training loss and perplexity:

Other mGPT - 1.3B models

• [🇦🇲 mGPT - 1.3B Armenian](https://huggingface.co/ai - forever/mGPT - 1.3B - armenian)
• [🇦🇿 mGPT - 1.3B Azerbaijan](https://huggingface.co/ai - forever/mGPT - 1.3B - azerbaijan)
• [🍯 mGPT - 1.3B Bashkir](https://huggingface.co/ai - forever/mGPT - 1.3B - bashkir)
• [🇧🇾 mGPT - 1.3B Belorussian](https://huggingface.co/ai - forever/mGPT - 1.3B - belorussian)
• [🇧🇬 mGPT - 1.3B Bulgarian](https://huggingface.co/ai - forever/mGPT - 1.3B - bulgarian)
• [🌞 mGPT - 1.3B Buryat](https://huggingface.co/ai - forever/mGPT - 1.3B - buryat)
• [🌳 mGPT - 1.3B Chuvash](https://huggingface.co/ai - forever/mGPT - 1.3B - chuvash)
• [🇬🇪 mGPT - 1.3B Georgian](https://huggingface.co/ai - forever/mGPT - 1.3B - georgian)
• [🌸 mGPT - 1.3B Kalmyk](https://huggingface.co/ai - forever/mGPT - 1.3B - kalmyk)
• [🇰🇬 mGPT - 1.3B Kirgiz](https://huggingface.co/ai - forever/mGPT - 1.3B - kirgiz)
• [🐻 mGPT - 1.3B Mari](https://huggingface.co/ai - forever/mGPT - 1.3B - mari)
• [🇲🇳 mGPT - 1.3B Mongol](https://huggingface.co/ai - forever/mGPT - 1.3B - mongol)
• [🐆 mGPT - 1.3B Ossetian](https://huggingface.co/ai - forever/mGPT - 1.3B - ossetian)
• [🇮🇷 mGPT - 1.3B Persian](https://huggingface.co/ai - forever/mGPT - 1.3B - persian)
• [🇷🇴 mGPT - 1.3B Romanian](https://huggingface.co/ai - forever/mGPT - 1.3B - romanian)
• [🇹🇯 mGPT - 1.3B Tajik](https://huggingface.co/ai - forever/mGPT - 1.3B - tajik)
• [☕ mGPT - 1.3B Tatar](https://huggingface.co/ai - forever/mGPT - 1.3B - tatar)
• [🇹🇲 mGPT - 1.3B Turkmen](https://huggingface.co/ai - forever/mGPT - 1.3B - turkmen)
• [🐎 mGPT - 1.3B Tuvan](https://huggingface.co/ai - forever/mGPT - 1.3B - tuvan)
• [🇺🇦 mGPT - 1.3B Ukranian](https://huggingface.co/ai - forever/mGPT - 1.3B - ukranian)
• [🇺🇿 mGPT - 1.3B Uzbek](https://huggingface.co/ai - forever/mGPT - 1.3B - uzbek)
• [💎 mGPT - 1.3B Yakut](https://huggingface.co/ai - forever/mGPT - 1.3B - yakut)

📄 License

This project is licensed under the MIT license.

💡 Usage Tip

If you find a bug or have additional data for training the model on your language, please provide us with feedback. The model will be improved over time, so stay tuned!