Mgpt 1.3B Uzbek

🚀 🇺🇿 Uzbek mGPT 1.3B

This is a language model designed specifically for the Uzbek language. As the name suggests, the model has 1.3 billion parameters.

Uzbek belongs to the Turkic language family. It is a highly rhythmic language spoken by approximately 32 million people. Here are some key facts about it:

1. It is the official language of Uzbekistan.
2. After Uzbekistan's independence, it transitioned from the Cyrillic script to the Latin script, though the Cyrillic script is still used by older generations.
3. Historically, it has been influenced by Persian and Arabic due to trade and Islamic scholarly traditions.

✨ Features

• Specifically tailored for the Uzbek language.
• Based on the powerful mGPT architecture.

📚 Documentation

🔧 Technical Details

This model is derived from the base mGPT-XL (1.3B) model. The base model was initially trained on 61 languages from 25 language families using Wikipedia and the C4 corpus.

We sourced additional data for 23 languages, most of which are considered minor, and decided to fine - tune the base model further. Uzbek mGPT 1.3B was trained for an additional 50,000 steps with a batch size of 4 and a context window of 2048 tokens on a single A100 GPU.

The final perplexity of this model on the validation set is 6.84.

Chart of the training loss and perplexity:

Other mGPT - 1.3B models

• 🇦🇲 mGPT - 1.3B Armenian
• 🇦🇿 mGPT - 1.3B Azerbaijan
• 🍯 mGPT - 1.3B Bashkir
• 🇧🇾 mGPT - 1.3B Belorussian
• 🇧🇬 mGPT - 1.3B Bulgarian
• 🌞 mGPT - 1.3B Buryat
• 🌳 mGPT - 1.3B Chuvash
• 🇬🇪 mGPT - 1.3B Georgian
• 🌸 mGPT - 1.3B Kalmyk
• 🇰🇿 mGPT - 1.3B Kazakh
• 🇰🇬 mGPT - 1.3B Kirgiz
• 🐻 mGPT - 1.3B Mari
• 🇲🇳 mGPT - 1.3B Mongol
• 🐆 mGPT - 1.3B Ossetian
• 🇮🇷 mGPT - 1.3B Persian
• 🇷🇴 mGPT - 1.3B Romanian
• 🇹🇯 mGPT - 1.3B Tajik
• ☕ mGPT - 1.3B Tatar
• 🇹🇲 mGPT - 1.3B Turkmen
• 🐎 mGPT - 1.3B Tuvan
• 🇺🇦 mGPT - 1.3B Ukranian
• 💎 mGPT - 1.3B Yakut

📄 License

This project is licensed under the MIT license.

💡 Usage Tip

If you find a bug or have additional data to train the model on your language, please provide us with feedback.

The model will be improved over time. Stay tuned!