đ đ mGPT 13B
A multilingual language model trained on 61 languages from 25 language families.
đ Quick Start
This is a multilingual language model trained on 61 languages from 25 language families (see the list below).
⨠Features
- Multilingual Support: Covers 61 languages from 25 language families.
- Large-scale Training: Pretrained on 600Gb of texts.
đĻ Installation
No installation steps provided in the original document.
đģ Usage Examples
No code examples provided in the original document.
đ Documentation
Dataset
The model was pretrained on 600Gb of texts, mostly from MC4 and Wikipedia. The training data was deduplicated. Text deduplication involves 64 - bit hashing of each text in the corpus to keep texts with a unique hash. Documents are also filtered based on their text compression rate using zlib4, and the most strongly and weakly compressing deduplicated texts are discarded.
Here is the table with the number of tokens for each language in the pretraining corpus on a logarithmic scale:
Languages
Afrikaans (af), Arabic (ar), Armenian (hy), Azerbaijani (az), Basque (eu), Bashkir (ba), Belarusian (be), Bengali (bn), Bulgarian (bg), Burmese (my), Buryat (bxr), Chuvash (cv), Danish (da), English (en), Estonian (et), Finnish (fi), French (fr), Georgian (ka), German (de), Greek (el), Hebrew (he), Hindi (hi), Hungarian (hu), Indonesian (id), Italian (it), Japanese (ja), Javanese (jv), Kalmyk (xal), Kazakh (kk), Korean (ko), Kyrgyz (ky), Latvian (lv), Lithuanian (lt), Malay (ms), Malayalam (ml), Marathi (mr), Mongolian (mn), Ossetian (os), Persian (fa), Polish (pl), Portuguese (pt), Romanian (ro), Russian (ru), Spanish (es), Swedish (sv), Swahili (sw), Tatar (tt), Telugu (te), Thai (th), Turkish (tr), Turkmen (tk), Tuvan (tyv), Ukrainian (uk), Uzbek (uz), Vietnamese (vi), Yakut (sax), Yoruba (yo)
By language family
| Language Family |
Languages |
| Afro - Asiatic |
Arabic (ar), Hebrew (he) |
| Austro - Asiatic |
Vietnamese (vi) |
| Austronesian |
Indonesian (id), Javanese (jv), Malay (ms), Tagalog (tl) |
| Baltic |
Latvian (lv), Lithuanian (lt) |
| Basque |
Basque (eu) |
| Dravidian |
Malayalam (ml), Tamil (ta), Telugu (te) |
| Indo - European (Armenian) |
Armenian (hy) |
| Indo - European (Indo - Aryan) |
Bengali (bn), Marathi (mr), Hindi (hi), Urdu (ur) |
| Indo - European (Germanic) |
Afrikaans (af), Danish (da), English (en), German (de), Swedish (sv) |
| Indo - European (Romance) |
French (fr), Italian (it), Portuguese (pt), Romanian (ro), Spanish (es) |
| Indo - European (Greek) |
Greek (el) |
| Indo - European (Iranian) |
Ossetian (os), Tajik (tg), Persian (fa) |
| Japonic |
Japanese (ja) |
| Kartvelian |
Georgian (ka) |
| Koreanic |
Korean (ko) |
| Kra - Dai |
Thai (th) |
| Mongolic |
Buryat (bxr), Kalmyk (xal), Mongolian (mn) |
| Niger - Congo |
Swahili (sw), Yoruba (yo) |
| Slavic |
Belarusian (be), Bulgarian (bg), Russian (ru), Ukrainian (uk), Polish (pl) |
| Sino - Tibetan |
Burmese (my) |
| Turkic (Karluk) |
Uzbek (uz) |
| Turkic (Kipchak) |
Bashkir (ba), Kazakh (kk), Kyrgyz (ky), Tatar (tt) |
| Turkic (Oghuz) |
Azerbaijani (az), Chuvash (cv), Turkish (tr), Turkmen (tk) |
| Turkic (Siberian) |
Tuvan (tyv), Yakut (sax) |
| Uralic |
Estonian (et), Finnish (fi), Hungarian (hu) |
đ§ Technical Details
The models are pretrained on 16 V100 GPUs for 600k training steps with a set of fixed hyperparameters: vocabulary size of 100k, context window of 2048, learning rate of 2eâ4, and batch size of 4.
The mGPT architecture is based on GPT - 3. We use the architecture description by Brown et al., the code base on GPT - 2 (Radford et al., 2019) in the HuggingFace library (Wolf et al., 2020) and Megatron - LM (Shoeybi et al., 2019).
đ Perplexity
The mGPT13B model achieves the best perplexities within the 2 - to - 10 score range for the majority of languages, including Dravidian (Malayalam, Tamil, Telugu), Indo - Aryan (Bengali, Hindi, Marathi), Slavic (Belarusian, Ukrainian, Russian, Bulgarian), Sino - Tibetan (Burmese), Kipchak (Bashkir, Kazakh) and others. Higher perplexities up to 20 are for only seven languages from different families.
Language - wise perplexity results
Family - wise perplexity results
The scores are averaged over the number of languages within each family.
đ License
The model is released under the MIT license.
%20--%3e%3cdefs%3e%3cstyle%3e%20.st0%20{%20fill:%20%23061b40;%20}%20.st1%20{%20fill:%20%23306af1;%20}%20.st2%20{%20fill:%20%235ce5cf;%20}%20%3c/style%3e%3c/defs%3e%3cg%3e%3cpath%20class='st0'%20d='M55,10.5h9v3h-9v9h12V7.5h-12v3ZM64,19.5h-6v-3h6v3Z'/%3e%3cpolygon%20class='st0'%20points='69%2016.5%2078%2016.5%2078%2019.5%2069%2019.5%2069%2022.5%2081%2022.5%2081%2013.5%2072%2013.5%2072%2010.5%2081%2010.5%2081%207.5%2069%207.5%2069%2016.5'/%3e%3cpolygon%20class='st0'%20points='95%2010.5%2095%207.5%2083%207.5%2083%2022.5%2095%2022.5%2095%2019.5%2086%2019.5%2086%2016.5%2095%2016.5%2095%2013.5%2086%2013.5%2086%2010.5%2095%2010.5'/%3e%3cpath%20class='st0'%20d='M40,1.5v21h11.6l1.4-1.4v-7.6h0c0,0-1.4-1.5-1.4-1.5l1.4-1.4V2.9l-1.4-1.4h-11.6ZM50,19.5h-7v-6h7v6ZM50,10.5h-7v-6h7v6Z'/%3e%3c/g%3e%3cpath%20class='st1'%20d='M23.1,24L14.7,4.8l-4.9,11.2h3.8l-1.8,4H3.3L12.1,0H2C.9,0,0,.9,0,2v20c0,1.1.9,2,2,2h21.1Z'/%3e%3cpath%20class='st2'%20d='M34,0h-16.8l10.6,24h6.2c1.1,0,2-.9,2-2V2C36,.9,35.1,0,34,0ZM32.5,20h-4V4h4v16Z'/%3e%3c/svg%3e)
