Muril Base Cased

🚀 MuRIL: Multilingual Representations for Indian Languages

MuRIL is a BERT model pre - trained on 17 Indian languages and their transliterated counterparts. It offers masked word prediction capabilities and an encoder on TFHub, providing valuable multilingual representation for Indian languages.

🚀 Quick Start

MuRIL is a BERT model pre - trained on 17 Indian languages and their transliterated counterparts. In this repository, we've released the pre - trained model with the MLM layer intact, which enables masked word predictions. We've also made the encoder available on TFHub, along with an additional pre - processing module that transforms raw text into the expected input format for the encoder. You can find more details about MuRIL in this paper.

✨ Features

• Multilingual Pretraining: The model uses a BERT base architecture pretrained from scratch using Wikipedia, Common Crawl, PMINDIA, and Dakshina corpora for 17 Indian languages.
• Modified Training Paradigm: Similar to multilingual BERT but with modifications. It includes translation and transliteration segment pairs in training and uses an exponent value of 0.3 for upsampling to enhance low - resource performance.

📚 Documentation

Overview

This model employs a BERT base architecture [1] that is pretrained from scratch. The training data sources are the Wikipedia [2], Common Crawl [3], PMINDIA [4], and Dakshina [5] corpora for 17 [6] Indian languages.

The training paradigm is similar to multilingual BERT, with the following modifications:

• Inclusion of Segment Pairs: We include translation and transliteration segment pairs in training.
• Upsampling Exponent: We use an exponent value of 0.3 (instead of 0.7) for upsampling, which has been shown to enhance low - resource performance [7].

Training

The MuRIL model is pre - trained on both monolingual and parallel segments:

• Monolingual Data: We utilize publicly available corpora from Wikipedia and Common Crawl for 17 Indian languages.
• Parallel Data:
  • Translated Data: We obtain translations of the above monolingual corpora using the Google NMT pipeline and also use the publicly available PMINDIA corpus.
  • Transliterated Data: We obtain transliterations of Wikipedia using the IndicTrans [8] library and also use the publicly available Dakshina dataset.

We use an exponent value of 0.3 to calculate duplication multiplier values for upsampling lower - resourced languages and set dupe factors accordingly. Note that we limit transliterated pairs to Wikipedia only.

The model was trained using a self - supervised masked language modeling task with whole word masking (maximum of 80 predictions). It was trained for 1000K steps, with a batch size of 4096 and a max sequence length of 512.

Trainable parameters

All parameters in the module are trainable, and fine - tuning all parameters is the recommended practice.

Uses & Limitations

This model is designed for a variety of downstream NLP tasks for Indian languages. It is trained on transliterated data, which is common in the Indian context. However, it may not perform well on languages other than the 17 Indian languages used in pretraining.

Evaluation

We provide the results of fine - tuning this model on a set of downstream tasks from the XTREME benchmark, evaluated on Indian language test - sets and their transliterated versions.

Results on XTREME benchmark datasets (in %)

Results on transliterated test - sets

References

[1]: Jacob Devlin, Ming - Wei Chang, Kenton Lee, Kristina Toutanova. BERT: Pre - training of Deep Bidirectional Transformers for Language Understanding. arXiv preprint arXiv:1810.04805, 2018. [2]: Wikipedia [3]: [Common Crawl](http://commoncrawl.org/the - data/) [4]: [PMINDIA](http://lotus.kuee.kyoto - u.ac.jp/WAT/indic - multilingual/index.html) [5]: [Dakshina](https://github.com/google - research - datasets/dakshina) [6]: Assamese (as), Bengali (bn), English (en), Gujarati (gu), Hindi (hi), Kannada (kn), Kashmiri (ks), Malayalam (ml), Marathi (mr), Nepali (ne), Oriya (or), Punjabi (pa), Sanskrit (sa), Sindhi (sd), Tamil (ta), Telugu (te) and Urdu (ur). [7]: Conneau, Alexis, et al. Unsupervised cross - lingual representation learning at scale. arXiv preprint arXiv:1911.02116 (2019). [8]: [IndicTrans](https://github.com/libindic/indic - trans) [9]: Hu, J., Ruder, S., Siddhant, A., Neubig, G., Firat, O., & Johnson, M. (2020). Xtreme: A massively multilingual multi - task benchmark for evaluating cross - lingual generalization. arXiv preprint arXiv:2003.11080. [10]: Fang, Y., Wang, S., Gan, Z., Sun, S., & Liu, J. (2020). FILTER: An Enhanced Fusion Method for Cross - lingual Language Understanding. arXiv preprint arXiv:2009.05166.

Citation

If you find MuRIL useful in your applications, please cite the following paper:

@misc{khanuja2021muril,
      title={MuRIL: Multilingual Representations for Indian Languages},
      author={Simran Khanuja and Diksha Bansal and Sarvesh Mehtani and Savya Khosla and Atreyee Dey and Balaji Gopalan and Dilip Kumar Margam and Pooja Aggarwal and Rajiv Teja Nagipogu and Shachi Dave and Shruti Gupta and Subhash Chandra Bose Gali and Vish Subramanian and Partha Talukdar},
      year={2021},
      eprint={2103.10730},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

Contact

Please mail your queries/feedback to muril - contact@google.com.

📄 License

This project is licensed under the apache - 2.0 license.