Xlmindic Base Uniscript Soham

🚀 XLMIndic Base Uniscript

This model is fine-tuned from this model on the Soham Bangla News Classification task, which is part of the IndicGLUE benchmark. It aims to provide effective language processing capabilities for Indo - Aryan languages.

Supported Languages

• as
• bn
• gu
• hi
• mr
• ne
• or
• pa
• si
• sa
• bpy
• mai
• bh
• gom

License

apache - 2.0

Datasets

• oscar

Tags

• multilingual
• albert
• xlmindic
• nlp
• indoaryan
• indicnlp
• iso15919
• transliteration
• text - classification

Widget Example

• text : 'cīnēra madhyāñcalē āraō ēkaṭi śaharēra bāsindārā ābāra gharabandī haẏē paṛēchēna. āja maṅgalabāra natuna karē lakaḍāuna–saṁkrānta bidhiniṣēdha jāri haōẏāra para gharē āṭakā paṛēchēna tām̐rā. karōnāra ati saṁkrāmaka natuna dharana amikranēra bistāra ṭhēkātē ēmana padakṣēpa niẏēchē kartr̥pakṣa. khabara bārtā saṁsthā ēēphapira.'

CO2 Emissions

🚀 Quick Start

This model is finetuned from this model on the Soham Bangla News Classification task, which is part of the IndicGLUE benchmark. Before pretraining this model, we transliterate the text to ISO - 15919 format using the Aksharamukha library. A demo of the Aksharamukha library is hosted here, where you can transliterate your text and use it on our model in the inference widget.

✨ Features

• Same Configuration as ALBERT Base v2: This model has the same configuration as the ALBERT Base v2 model. Specifically, it has 12 repeating layers, a 128 - embedding dimension, a 768 - hidden dimension, 12 attention heads, 11M parameters, and a 512 - sequence length.
• Transliteration Support: It takes in ISO - 15919 transliterated text, which helps the model learn better cross - lingual representations when dealing with languages that share vocabularies but use different writing scripts.

📦 Installation

To use this model, you will need to first install the Aksharamukha library.

pip install aksharamukha

💻 Usage Examples

Basic Usage

>>> from aksharamukha import transliterate
>>> text = "चूंकि मानव परिवार के सभी सदस्यों के जन्मजात गौरव और समान तथा अविच्छिन्न अधिकार की स्वीकृति ही विश्व-शान्ति, न्याय और स्वतन्त्रता की बुनियाद है"
>>> transliterated_text = transliterate.process('autodetect', 'ISO', text)
>>> transliterated_text
"cūṁki mānava parivāra kē sabhī sadasyōṁ kē janmajāta gaurava aura samāna tathā avicchinna adhikāra kī svīkr̥ti hī viśva-śānti, nyāya aura svatantratā kī buniyāda hai"

Advanced Usage

>>> from transformers import pipeline
>>> from aksharamukha import transliterate
>>> unmasker = pipeline('fill-mask', model='ibraheemmoosa/xlmindic-base-uniscript')
>>> text = "রবীন্দ্রনাথ ঠাকুর এফআরএএস (৭ মে ১৮৬১ - ৭ আগস্ট ১৯৪১; ২৫ বৈশাখ ১২৬৮ - ২২ শ্রাবণ ১৩৪৮ বঙ্গাব্দ) ছিলেন অগ্রণী বাঙালি [MASK], ঔপন্যাসিক, সংগীতস্রষ্টা, নাট্যকার, চিত্রকর, ছোটগল্পকার, প্রাবন্ধিক, অভিনেতা, কণ্ঠশিল্পী ও দার্শনিক। ১৯১৩ সালে গীতাঞ্জলি কাব্যগ্রন্থের ইংরেজি অনুবাদের জন্য তিনি এশীয়দের মধ্যে সাহিত্যে প্রথম নোবেল পুরস্কার লাভ করেন।"
>>> transliterated_text = transliterate.process('Bengali', 'ISO', text)
>>> transliterated_text
'rabīndranātha ṭhākura ēphaāraēēsa (7 mē 1861 - 7 āgasṭa 1941; 25 baiśākha 1268 - 22 śrābaṇa 1348 baṅgābda) chilēna agraṇī bāṅāli [MASK], aupanyāsika, saṁgītasraṣṭā, nāṭyakāra, citrakara, chōṭagalpakāra, prābandhika, abhinētā, kaṇṭhaśilpī ō dārśanika. 1913 sālē gītāñjali kābyagranthēra iṁrēji anubādēra janya tini ēśīẏadēra madhyē sāhityē prathama [MASK] puraskāra lābha karēna.'
>>> unmasker(transliterated_text)
[{'score': 0.39705055952072144,
  'token': 1500,
  'token_str': 'abhinētā',
  'sequence': 'rabīndranātha ṭhākura ēphaāraēēsa (7 mē 1861 - 7 āgasṭa 1941; 25 baiśākha 1268 - 22 śrābaṇa 1348 baṅgābda) chilēna agraṇī bāṅāli abhinētā, aupanyāsika, saṁgītasraṣṭā, nāṭyakāra, citrakara, chōṭagalpakāra, prābandhika, abhinētā, kaṇṭhaśilpī ō dārśanika. 1913 sālē gītāñjali kābyagranthēra iṁrēji anubādēra janya tini ēśīẏadēra madhyē sāhityē prathama nōbēla puraskāra lābha karēna.'},
 {'score': 0.20499080419540405,
  'token': 3585,
  'token_str': 'kabi',
  'sequence': 'rabīndranātha ṭhākura ēphaāraēēsa (7 mē 1861 - 7 āgasṭa 1941; 25 baiśākha 1268 - 22 śrābaṇa 1348 baṅgābda) chilēna agraṇī bāṅāli kabi, aupanyāsika, saṁgītasraṣṭā, nāṭyakāra, citrakara, chōṭagalpakāra, prābandhika, abhinētā, kaṇṭhaśilpī ō dārśanika. 1913 sālē gītāñjali kābyagranthēra iṁrēji anubādēra janya tini ēśīẏadēra madhyē sāhityē prathama nōbēla puraskāra lābha karēna.'},
 {'score': 0.1314290314912796,
  'token': 15402,
  'token_str': 'rājanētā',
  'sequence': 'rabīndranātha ṭhākura ēphaāraēēsa (7 mē 1861 - 7 āgasṭa 1941; 25 baiśākha 1268 - 22 śrābaṇa 1348 baṅgābda) chilēna agraṇī bāṅāli rājanētā, aupanyāsika, saṁgītasraṣṭā, nāṭyakāra, citrakara, chōṭagalpakāra, prābandhika, abhinētā, kaṇṭhaśilpī ō dārśanika. 1913 sālē gītāñjali kābyagranthēra iṁrēji anubādēra janya tini ēśīẏadēra madhyē sāhityē prathama nōbēla puraskāra lābha karēna.'},
 {'score': 0.060830358415842056,
  'token': 3212,
  'token_str': 'kalākāra',
  'sequence': 'rabīndranātha ṭhākura ēphaāraēēsa (7 mē 1861 - 7 āgasṭa 1941; 25 baiśākha 1268 - 22 śrābaṇa 1348 baṅgābda) chilēna agraṇī bāṅāli kalākāra, aupanyāsika, saṁgītasraṣṭā, nāṭyakāra, citrakara, chōṭagalpakāra, prābandhika, abhinētā, kaṇṭhaśilpī ō dārśanika. 1913 sālē gītāñjali kābyagranthēra iṁrēji anubādēra janya tini ēśīẏadēra madhyē sāhityē prathama nōbēla puraskāra lābha karēna.'},
 {'score': 0.035522934049367905,
  'token': 11586,
  'token_str': 'sāhityakāra',
  'sequence': 'rabīndranātha ṭhākura ēphaāraēēsa (7 mē 1861 - 7 āgasṭa 1941; 25 baiśākha 1268 - 22 śrābaṇa 1348 baṅgābda) chilēna agraṇī bāṅāli sāhityakāra, aupanyāsika, saṁgītasraṣṭā, nāṭyakāra, citrakara, chōṭagalpakāra, prābandhika, abhinētā, kaṇṭhaśilpī ō dārśanika. 1913 sālē gītāñjali kābyagranthēra iṁrēji anubādēra janya tini ēśīẏadēra madhyē sāhityē prathama nōbēla puraskāra lābha karēna.'}]

📚 Documentation

Model Description

This model has the same configuration as the ALBERT Base v2 model. Specifically, this model has the following configuration:

• 12 repeating layers
• 128 embedding dimension
• 768 hidden dimension
• 12 attention heads
• 11M parameters
• 512 sequence length

Training Data

This model was fine - tuned on the Soham dataset, which is part of the IndicGLUE benchmark.

Transliteration

The unique component of this model is that it takes in ISO - 15919 transliterated text.

The motivation behind this is as follows. When two languages share vocabularies, a machine learning model can exploit that to learn good cross - lingual representations. However, if these two languages use different writing scripts, it is difficult for a model to make the connection. Thus, if we can write the two languages in a single script, it is easier for the model to learn good cross - lingual representation.

For many of the scripts currently in use, there are standard transliteration schemes to convert to the Latin script. In particular, for the Indic scripts, the ISO - 15919 transliteration scheme is designed to consistently transliterate texts written in different Indic scripts to the Latin script.

An example of ISO - 15919 transliteration for a piece of Bangla text is the following:

Original: "রবীন্দ্রনাথ ঠাকুর এফআরএএস (৭ মে ১৮৬১ - ৭ আগস্ট ১৯৪১; ২৫ বৈশাখ ১২৬৮ - ২২ শ্রাবণ ১৩৪৮ বঙ্গাব্দ) ছিলেন অগ্রণী বাঙালি কবি, ঔপন্যাসিক, সংগীতস্রষ্টা, নাট্যকার, চিত্রকর, ছোটগল্পকার, প্রাবন্ধিক, অভিনেতা, কণ্ঠশিল্পী ও দার্শনিক।"

Transliterated: 'rabīndranātha ṭhākura ēphaāraēēsa (7 mē 1861 - 7 āgasṭa 1941; 25 baiśākha 1268 - 22 śrābaṇa 1348 baṅgābda) chilēna agraṇī bāṅāli kabi, aupanyāsika, saṁgītasraṣṭā, nāṭyakāra, citrakara, chōṭagalpakāra, prābandhika, abhinētā, kaṇṭhaśilpī ō dārśanika.'

Another example for a piece of Hindi text is the following:

Original: "चूंकि मानव परिवार के सभी सदस्यों के जन्मजात गौरव और समान तथा अविच्छिन्न अधिकार की स्वीकृति ही विश्व-शान्ति, न्याय और स्वतन्त्रता की बुनियाद है"

Transliterated: "cūṁki mānava parivāra kē sabhī sadasyōṁ kē janmajāta gaurava aura samāna tathā avicchinna adhikāra kī svīkr̥ti hī viśva-śānti, nyāya aura svatantratā kī buniyāda hai"

Training Procedure

Preprocessing

The texts are transliterated to ISO - 15919 format using the Aksharamukha library. Then these are tokenized using SentencePiece and a vocabulary size of 50,000.

Training

The model was trained for 8 epochs with a batch size of 16 and a learning rate of 2e - 5.

Evaluation Results

See results specific to Soham in the following table.

Intended Uses & Limitations

This model is pretrained on Indo - Aryan languages. Thus it is intended to be used for downstream tasks on these languages. However, since Dravidian languages such as Malayalam, Telegu, Kannada etc share a lot of vocabulary with the Indo - Aryan languages, this model can potentially be used on those languages too (after transliterating the text to ISO - 15919).

You can use the raw model for either masked language modeling or next sentence prediction, but it's mostly intended to be fine - tuned on a downstream task. See the model hub to look for fine - tuned versions on a task that interests you. Note that this model is primarily aimed at being fine - tuned on tasks that use the whole sentence (potentially masked) to make decisions, such as sequence classification, token classification or question answering. For tasks such as text generation you should look at models like GPT2.

Limitations and Bias

Even though we pretrain on a comparatively large multilingual corpus, the model may exhibit harmful gender, ethnic and political bias. If you fine - tune this model on a task where these issues are important, you should take special care when relying on the model to make decisions.

🔧 Technical Details

The key technical details of this model include the pre - processing step of transliterating text to ISO - 15919 format using the Aksharamukha library. This step helps the model better handle languages with different scripts but shared vocabularies. During training, the model uses SentencePiece for tokenization with a vocabulary size of 50,000. It was trained for 8 epochs with a batch size of 16 and a learning rate of 2e - 5.

📄 License

This project is licensed under the apache - 2.0 license.

Contact

Feel free to contact us if you have any ideas or if you want to know more about our models.

• Ibraheem Muhammad Moosa (ibraheemmoosa1347@gmail.com)
• Mahmud Elahi Akhter (mahmud.akhter01@northsouth.edu)
• Ashfia Binte Habib

BibTeX entry and citation info

Coming soon!