🚀 AI-translator-eng-to-9ja
This is a 418-million parameter translation model designed to translate from English into Yoruba, Igbo, and Hausa. Trained on a dataset of 1,500,000 sentences (500,000 for each language), it offers high-quality translations for these languages. The model aims to build a system that simplifies communication with LLMs using Igbo, Hausa, and Yoruba.
✨ Features
- Multilingual Translation: Translates from English to Yoruba, Igbo, and Hausa.
- High-Quality Output: Trained on a large dataset for accurate translations.
- LLM Communication: Facilitates communication with LLMs in local languages.
📦 Installation
No specific installation steps are provided in the original document.
💻 Usage Examples
Basic Usage
from transformers import M2M100ForConditionalGeneration, M2M100Tokenizer
import huggingface_hub
huggingface_hub.login()
model = M2M100ForConditionalGeneration.from_pretrained("HelpMumHQ/AI-translator-eng-to-9ja")
tokenizer = M2M100Tokenizer.from_pretrained("HelpMumHQ/AI-translator-eng-to-9ja")
eng_text="Healthcare is an important field in virtually every society because it directly affects the well-being and quality of life of individuals. It encompasses a wide range of services and professions, including preventive care, diagnosis, treatment, and management of diseases and conditions."
tokenizer.src_lang = "en"
tokenizer.tgt_lang = "ig"
encoded_eng = tokenizer(eng_text, return_tensors="pt")
generated_tokens = model.generate(**encoded_eng, forced_bos_token_id=tokenizer.get_lang_id("ig"))
tokenizer.batch_decode(generated_tokens, skip_special_tokens=True)
eng_text="Healthcare is an important field in virtually every society because it directly affects the well-being and quality of life of individuals. It encompasses a wide range of services and professions, including preventive care, diagnosis, treatment, and management of diseases and conditions. Effective healthcare systems aim to improve health outcomes, reduce the incidence of illness, and ensure that individuals have access to necessary medical services."
tokenizer.src_lang = "en"
tokenizer.tgt_lang = "yo"
encoded_eng = tokenizer(eng_text, return_tensors="pt")
generated_tokens = model.generate(**encoded_eng, forced_bos_token_id=tokenizer.get_lang_id("yo"))
tokenizer.batch_decode(generated_tokens, skip_special_tokens=True)
eng_text="Healthcare is an important field in virtually every society because it directly affects the well-being and quality of life of individuals. It encompasses a wide range of services and professions, including preventive care, diagnosis, treatment, and management of diseases and conditions. Effective healthcare systems aim to improve health outcomes, reduce the incidence of illness, and ensure that individuals have access to necessary medical services."
tokenizer.src_lang = "en"
tokenizer.tgt_lang = "ha"
encoded_eng = tokenizer(eng_text, return_tensors="pt")
generated_tokens = model.generate(**encoded_eng, forced_bos_token_id=tokenizer.get_lang_id("ha"))
tokenizer.batch_decode(generated_tokens, skip_special_tokens=True)
📚 Documentation
Model Details
| Property |
Details |
| Languages Supported |
Source Language: English; Target Languages: Yoruba, Igbo, Hausa |
| Model Type |
418 Million parameter translation model |
| Training Data |
A dataset of 1,500,000 translation pairs from open - source parallel corpora and curated datasets for Yoruba, Igbo, and Hausa |
Supported Language Codes
- English:
en
- Yoruba:
yo
- Igbo:
ig
- Hausa:
ha
Training Dataset
The training dataset consists of 1,500,000 translation pairs, sourced from a combination of open - source parallel corpora and curated datasets specific to Yoruba, Igbo, and Hausa.
Limitations
- Performance may vary depending on the complexity and domain of the text, even though the model performs well in English - to - Yoruba, Igbo, and Hausa translations.
- Translation quality may decline for extremely long sentences or ambiguous contexts.
Training Hyperparameters
The following hyperparameters were used during training:
- learning_rate: 2e - 05
- train_batch_size: 64
- eval_batch_size: 64
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon = 1e - 08
- lr_scheduler_type: linear
- num_epochs: 3
Framework Versions
- Transformers 4.44.2
- Pytorch 2.4.0+cu121
- Datasets 2.21.0
- Tokenizers 0.19.1
📄 License
This model is licensed under the MIT license.
%20--%3e%3cdefs%3e%3cstyle%3e%20.st0%20{%20fill:%20%23061b40;%20}%20.st1%20{%20fill:%20%23306af1;%20}%20.st2%20{%20fill:%20%235ce5cf;%20}%20%3c/style%3e%3c/defs%3e%3cg%3e%3cpath%20class='st0'%20d='M55,10.5h9v3h-9v9h12V7.5h-12v3ZM64,19.5h-6v-3h6v3Z'/%3e%3cpolygon%20class='st0'%20points='69%2016.5%2078%2016.5%2078%2019.5%2069%2019.5%2069%2022.5%2081%2022.5%2081%2013.5%2072%2013.5%2072%2010.5%2081%2010.5%2081%207.5%2069%207.5%2069%2016.5'/%3e%3cpolygon%20class='st0'%20points='95%2010.5%2095%207.5%2083%207.5%2083%2022.5%2095%2022.5%2095%2019.5%2086%2019.5%2086%2016.5%2095%2016.5%2095%2013.5%2086%2013.5%2086%2010.5%2095%2010.5'/%3e%3cpath%20class='st0'%20d='M40,1.5v21h11.6l1.4-1.4v-7.6h0c0,0-1.4-1.5-1.4-1.5l1.4-1.4V2.9l-1.4-1.4h-11.6ZM50,19.5h-7v-6h7v6ZM50,10.5h-7v-6h7v6Z'/%3e%3c/g%3e%3cpath%20class='st1'%20d='M23.1,24L14.7,4.8l-4.9,11.2h3.8l-1.8,4H3.3L12.1,0H2C.9,0,0,.9,0,2v20c0,1.1.9,2,2,2h21.1Z'/%3e%3cpath%20class='st2'%20d='M34,0h-16.8l10.6,24h6.2c1.1,0,2-.9,2-2V2C36,.9,35.1,0,34,0ZM32.5,20h-4V4h4v16Z'/%3e%3c/svg%3e)
