đ eng-iir Translation Model
This project focuses on the translation from English to Indo - Iranian languages. It provides a transformer - based model with specific pre - processing steps and offers detailed benchmarks and system information.
đ Quick Start
- The model is of the
transformer type.
- It requires pre - processing with normalization and SentencePiece (spm32k,spm32k).
- A sentence initial language token in the form of
>>id<< (where id is a valid target language ID) is necessary.
⨠Features
- Multi - target Translation: Capable of translating English into multiple Indo - Iranian languages, including
asm, awa, ben, etc.
- Benchmarked Performance: Provides BLEU and chr - F scores on various test sets to evaluate the model's performance.
đĻ Installation
No specific installation steps are provided in the original document.
đģ Usage Examples
No code examples are provided in the original document.
đ Documentation
eng - iir Model Details
- Source Group: English
- Target Group: Indo - Iranian languages
- OPUS Readme: eng - iir
- Model: transformer
- Source Language(s): eng
- Target Language(s): asm awa ben bho gom guj hif_Latn hin jdt_Cyrl kur_Arab kur_Latn mai mar npi ori oss pan_Guru pes pes_Latn pes_Thaa pnb pus rom san_Deva sin snd_Arab tgk_Cyrl tly_Latn urd zza
- Pre - processing: normalization + SentencePiece (spm32k,spm32k)
- Download Original Weights: opus2m - 2020 - 08 - 01.zip
- Test Set Translations: opus2m - 2020 - 08 - 01.test.txt
- Test Set Scores: opus2m - 2020 - 08 - 01.eval.txt
Benchmarks
| Testset |
BLEU |
chr - F |
| newsdev2014 - enghin.eng.hin |
6.7 |
0.326 |
| newsdev2019 - engu - engguj.eng.guj |
6.0 |
0.283 |
| newstest2014 - hien - enghin.eng.hin |
10.4 |
0.353 |
| newstest2019 - engu - engguj.eng.guj |
6.6 |
0.282 |
| Tatoeba - test.eng - asm.eng.asm |
2.7 |
0.249 |
| Tatoeba - test.eng - awa.eng.awa |
0.4 |
0.122 |
| Tatoeba - test.eng - ben.eng.ben |
15.3 |
0.459 |
| Tatoeba - test.eng - bho.eng.bho |
3.7 |
0.161 |
| Tatoeba - test.eng - fas.eng.fas |
3.4 |
0.227 |
| Tatoeba - test.eng - guj.eng.guj |
18.5 |
0.365 |
| Tatoeba - test.eng - hif.eng.hif |
1.0 |
0.064 |
| Tatoeba - test.eng - hin.eng.hin |
17.0 |
0.461 |
| Tatoeba - test.eng - jdt.eng.jdt |
3.9 |
0.122 |
| Tatoeba - test.eng - kok.eng.kok |
5.5 |
0.059 |
| Tatoeba - test.eng - kur.eng.kur |
4.0 |
0.125 |
| Tatoeba - test.eng - lah.eng.lah |
0.3 |
0.008 |
| Tatoeba - test.eng - mai.eng.mai |
9.3 |
0.445 |
| Tatoeba - test.eng - mar.eng.mar |
20.7 |
0.473 |
| Tatoeba - test.eng.multi |
13.7 |
0.392 |
| Tatoeba - test.eng - nep.eng.nep |
0.6 |
0.060 |
| Tatoeba - test.eng - ori.eng.ori |
2.4 |
0.193 |
| Tatoeba - test.eng - oss.eng.oss |
2.1 |
0.174 |
| Tatoeba - test.eng - pan.eng.pan |
9.7 |
0.355 |
| Tatoeba - test.eng - pus.eng.pus |
1.0 |
0.126 |
| Tatoeba - test.eng - rom.eng.rom |
1.3 |
0.230 |
| Tatoeba - test.eng - san.eng.san |
1.3 |
0.101 |
| Tatoeba - test.eng - sin.eng.sin |
11.7 |
0.384 |
| Tatoeba - test.eng - snd.eng.snd |
2.8 |
0.180 |
| Tatoeba - test.eng - tgk.eng.tgk |
8.1 |
0.353 |
| Tatoeba - test.eng - tly.eng.tly |
0.5 |
0.015 |
| Tatoeba - test.eng - urd.eng.urd |
12.3 |
0.409 |
| Tatoeba - test.eng - zza.eng.zza |
0.5 |
0.025 |
System Info
| Property |
Details |
| hf_name |
eng - iir |
| source_languages |
eng |
| target_languages |
iir |
| opus_readme_url |
https://github.com/Helsinki-NLP/Tatoeba-Challenge/tree/master/models/eng-iir/README.md |
| original_repo |
Tatoeba - Challenge |
| tags |
['translation'] |
| languages |
['en', 'bn', 'or', 'gu', 'mr', 'ur', 'hi', 'ps', 'os', 'as', 'si', 'iir'] |
| src_constituents |
{'eng'} |
| tgt_constituents |
{'pnb', 'gom', 'ben', 'hif_Latn', 'ori', 'guj', 'pan_Guru', 'snd_Arab', 'npi', 'mar', 'urd', 'pes', 'bho', 'kur_Arab', 'tgk_Cyrl', 'hin', 'kur_Latn', 'pes_Thaa', 'pus', 'san_Deva', 'oss', 'tly_Latn', 'jdt_Cyrl', 'asm', 'zza', 'rom', 'mai', 'pes_Latn', 'awa', 'sin'} |
| src_multilingual |
False |
| tgt_multilingual |
True |
| prepro |
normalization + SentencePiece (spm32k,spm32k) |
| url_model |
https://object.pouta.csc.fi/Tatoeba-MT-models/eng-iir/opus2m-2020-08-01.zip |
| url_test_set |
https://object.pouta.csc.fi/Tatoeba-MT-models/eng-iir/opus2m-2020-08-01.test.txt |
| src_alpha3 |
eng |
| tgt_alpha3 |
iir |
| short_pair |
en - iir |
| chrF2_score |
0.392 |
| bleu |
13.7 |
| brevity_penalty |
1.0 |
| ref_len |
63351.0 |
| src_name |
English |
| tgt_name |
Indo - Iranian languages |
| train_date |
2020 - 08 - 01 |
| src_alpha2 |
en |
| tgt_alpha2 |
iir |
| prefer_old |
False |
| long_pair |
eng - iir |
| helsinki_git_sha |
480fcbe0ee1bf4774bcbe6226ad9f58e63f6c535 |
| transformers_git_sha |
2207e5d8cb224e954a7cba69fa4ac2309e9ff30b |
| port_machine |
brutasse |
| port_time |
2020 - 08 - 21 - 14:41 |
đ License
The project is licensed under the apache - 2.0 license.
%20--%3e%3cdefs%3e%3cstyle%3e%20.st0%20{%20fill:%20%23061b40;%20}%20.st1%20{%20fill:%20%23306af1;%20}%20.st2%20{%20fill:%20%235ce5cf;%20}%20%3c/style%3e%3c/defs%3e%3cg%3e%3cpath%20class='st0'%20d='M55,10.5h9v3h-9v9h12V7.5h-12v3ZM64,19.5h-6v-3h6v3Z'/%3e%3cpolygon%20class='st0'%20points='69%2016.5%2078%2016.5%2078%2019.5%2069%2019.5%2069%2022.5%2081%2022.5%2081%2013.5%2072%2013.5%2072%2010.5%2081%2010.5%2081%207.5%2069%207.5%2069%2016.5'/%3e%3cpolygon%20class='st0'%20points='95%2010.5%2095%207.5%2083%207.5%2083%2022.5%2095%2022.5%2095%2019.5%2086%2019.5%2086%2016.5%2095%2016.5%2095%2013.5%2086%2013.5%2086%2010.5%2095%2010.5'/%3e%3cpath%20class='st0'%20d='M40,1.5v21h11.6l1.4-1.4v-7.6h0c0,0-1.4-1.5-1.4-1.5l1.4-1.4V2.9l-1.4-1.4h-11.6ZM50,19.5h-7v-6h7v6ZM50,10.5h-7v-6h7v6Z'/%3e%3c/g%3e%3cpath%20class='st1'%20d='M23.1,24L14.7,4.8l-4.9,11.2h3.8l-1.8,4H3.3L12.1,0H2C.9,0,0,.9,0,2v20c0,1.1.9,2,2,2h21.1Z'/%3e%3cpath%20class='st2'%20d='M34,0h-16.8l10.6,24h6.2c1.1,0,2-.9,2-2V2C36,.9,35.1,0,34,0ZM32.5,20h-4V4h4v16Z'/%3e%3c/svg%3e)
