đ eng - inc Translation Model
This is a translation model designed to translate from English to various Indic languages, offering a wide - range of language support and performance benchmarks.
đ Quick Start
The eng - inc model is a powerful tool for translating English to Indic languages. Here are some key resources to get you started:
- OPUS Readme: [eng - inc](https://github.com/Helsinki - NLP/Tatoeba - Challenge/tree/master/models/eng - inc/README.md)
- Download Original Weights: [opus2m - 2020 - 08 - 01.zip](https://object.pouta.csc.fi/Tatoeba - MT - models/eng - inc/opus2m - 2020 - 08 - 01.zip)
- Test Set Translations: [opus2m - 2020 - 08 - 01.test.txt](https://object.pouta.csc.fi/Tatoeba - MT - models/eng - inc/opus2m - 2020 - 08 - 01.test.txt)
- Test Set Scores: [opus2m - 2020 - 08 - 01.eval.txt](https://object.pouta.csc.fi/Tatoeba - MT - models/eng - inc/opus2m - 2020 - 08 - 01.eval.txt)
⨠Features
- Language Support: Supports multiple Indic languages including Assamese (
asm), Awadhi (awa), Bengali (ben), Bhojpuri (bho), etc.
- Model Type: Utilizes a
transformer model architecture.
- Pre - processing: Applies normalization and SentencePiece (
spm32k, spm32k) for pre - processing.
- Language Token Requirement: A sentence initial language token in the form of
>>id<< (where id is a valid target language ID) is required.
đ Documentation
Model Details
| Property |
Details |
| Model Type |
transformer |
| Source Languages |
eng |
| Target Languages |
asm, awa, ben, bho, gom, guj, hif_Latn, hin, mai, mar, npi, ori, pan_Guru, pnb, rom, san_Deva, sin, snd_Arab, urd |
| Pre - processing |
normalization + SentencePiece (spm32k, spm32k) |
Benchmarks
| testset |
BLEU |
chr - F |
| newsdev2014 - enghin.eng.hin |
8.2 |
0.342 |
| newsdev2019 - engu - engguj.eng.guj |
6.5 |
0.293 |
| newstest2014 - hien - enghin.eng.hin |
11.4 |
0.364 |
| newstest2019 - engu - engguj.eng.guj |
7.2 |
0.296 |
| Tatoeba - test.eng - asm.eng.asm |
2.7 |
0.277 |
| Tatoeba - test.eng - awa.eng.awa |
0.5 |
0.132 |
| Tatoeba - test.eng - ben.eng.ben |
16.7 |
0.470 |
| Tatoeba - test.eng - bho.eng.bho |
4.3 |
0.227 |
| Tatoeba - test.eng - guj.eng.guj |
17.5 |
0.373 |
| Tatoeba - test.eng - hif.eng.hif |
0.6 |
0.028 |
| Tatoeba - test.eng - hin.eng.hin |
17.7 |
0.469 |
| Tatoeba - test.eng - kok.eng.kok |
1.7 |
0.000 |
| Tatoeba - test.eng - lah.eng.lah |
0.3 |
0.028 |
| Tatoeba - test.eng - mai.eng.mai |
15.6 |
0.429 |
| Tatoeba - test.eng - mar.eng.mar |
21.3 |
0.477 |
| Tatoeba - test.eng.multi |
17.3 |
0.448 |
| Tatoeba - test.eng - nep.eng.nep |
0.8 |
0.081 |
| Tatoeba - test.eng - ori.eng.ori |
2.2 |
0.208 |
| Tatoeba - test.eng - pan.eng.pan |
8.0 |
0.347 |
| Tatoeba - test.eng - rom.eng.rom |
0.4 |
0.197 |
| Tatoeba - test.eng - san.eng.san |
0.5 |
0.108 |
| Tatoeba - test.eng - sin.eng.sin |
9.1 |
0.364 |
| Tatoeba - test.eng - snd.eng.snd |
4.4 |
0.284 |
| Tatoeba - test.eng - urd.eng.urd |
13.3 |
0.423 |
System Info
- hf_name: eng - inc
- source_languages: eng
- target_languages: inc
- opus_readme_url: https://github.com/Helsinki - NLP/Tatoeba - Challenge/tree/master/models/eng - inc/README.md
- original_repo: Tatoeba - Challenge
- tags: ['translation']
- languages: ['en', 'bn', 'or', 'gu', 'mr', 'ur', 'hi', 'as', 'si', 'inc']
- src_constituents: {'eng'}
- tgt_constituents: {'pnb', 'gom', 'ben', 'hif_Latn', 'ori', 'guj', 'pan_Guru', 'snd_Arab', 'npi', 'mar', 'urd', 'bho', 'hin', 'san_Deva', 'asm', 'rom', 'mai', 'awa', 'sin'}
- src_multilingual: False
- tgt_multilingual: True
- prepro: normalization + SentencePiece (spm32k, spm32k)
- url_model: https://object.pouta.csc.fi/Tatoeba - MT - models/eng - inc/opus2m - 2020 - 08 - 01.zip
- url_test_set: https://object.pouta.csc.fi/Tatoeba - MT - models/eng - inc/opus2m - 2020 - 08 - 01.test.txt
- src_alpha3: eng
- tgt_alpha3: inc
- short_pair: en - inc
- chrF2_score: 0.44799999999999995
- bleu: 17.3
- brevity_penalty: 1.0
- ref_len: 59917.0
- src_name: English
- tgt_name: Indic languages
- train_date: 2020 - 08 - 01
- src_alpha2: en
- tgt_alpha2: inc
- prefer_old: False
- long_pair: eng - inc
- helsinki_git_sha: 480fcbe0ee1bf4774bcbe6226ad9f58e63f6c535
- transformers_git_sha: 2207e5d8cb224e954a7cba69fa4ac2309e9ff30b
- port_machine: brutasse
- port_time: 2020 - 08 - 21 - 14:41
đ License
This project is licensed under the apache - 2.0 license.
%20--%3e%3cdefs%3e%3cstyle%3e%20.st0%20{%20fill:%20%23061b40;%20}%20.st1%20{%20fill:%20%23306af1;%20}%20.st2%20{%20fill:%20%235ce5cf;%20}%20%3c/style%3e%3c/defs%3e%3cg%3e%3cpath%20class='st0'%20d='M55,10.5h9v3h-9v9h12V7.5h-12v3ZM64,19.5h-6v-3h6v3Z'/%3e%3cpolygon%20class='st0'%20points='69%2016.5%2078%2016.5%2078%2019.5%2069%2019.5%2069%2022.5%2081%2022.5%2081%2013.5%2072%2013.5%2072%2010.5%2081%2010.5%2081%207.5%2069%207.5%2069%2016.5'/%3e%3cpolygon%20class='st0'%20points='95%2010.5%2095%207.5%2083%207.5%2083%2022.5%2095%2022.5%2095%2019.5%2086%2019.5%2086%2016.5%2095%2016.5%2095%2013.5%2086%2013.5%2086%2010.5%2095%2010.5'/%3e%3cpath%20class='st0'%20d='M40,1.5v21h11.6l1.4-1.4v-7.6h0c0,0-1.4-1.5-1.4-1.5l1.4-1.4V2.9l-1.4-1.4h-11.6ZM50,19.5h-7v-6h7v6ZM50,10.5h-7v-6h7v6Z'/%3e%3c/g%3e%3cpath%20class='st1'%20d='M23.1,24L14.7,4.8l-4.9,11.2h3.8l-1.8,4H3.3L12.1,0H2C.9,0,0,.9,0,2v20c0,1.1.9,2,2,2h21.1Z'/%3e%3cpath%20class='st2'%20d='M34,0h-16.8l10.6,24h6.2c1.1,0,2-.9,2-2V2C36,.9,35.1,0,34,0ZM32.5,20h-4V4h4v16Z'/%3e%3c/svg%3e)
