đ HPLT MT release v1.0
This repository offers a translation model for Arabic-English, trained using OPUS and HPLT data. It's available in both Marian and Hugging Face formats.
đ Quick Start
This repository contains a translation model for Arabic-English, trained with OPUS and HPLT data. It is available in both Marian and Hugging Face formats.
⨠Features
- Trained on OPUS and HPLT data for Arabic-English translation.
- Available in both Marian and Hugging Face formats, ensuring compatibility with different frameworks.
đĻ Installation
The model has been trained with MarianNMT and the weights are in the Marian format. It has also been converted into the Hugging Face format for compatibility with transformers.
Using Marian
To run inference with MarianNMT, refer to the Inference/Decoding/Translation section of our GitHub repository. You will need the model file model.npz.best-chrf.npz and the vocabulary file model.ar-en.spm from this repository.
Using transformers
We have also converted this model to the Hugging Face format. Note that due to a known issue in weight conversion, the checkpoint cannot work with transformer versions <4.26 or >4.30. We tested and suggest pip install transformers==4.28.
đģ Usage Examples
Basic Usage
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
tokenizer = AutoTokenizer.from_pretrained("HPLT/translate-ar-en-v1.0-hplt_opus")
model = AutoModelForSeq2SeqLM.from_pretrained("HPLT/translate-ar-en-v1.0-hplt_opus")
inputs = ["Input goes here.", "Make sure the language is right."]
batch_tokenized = tokenizer(inputs, return_tensors="pt", padding=True)
model_output = model.generate(
**batch_tokenized, num_beams=6, max_new_tokens=512
)
batch_detokenized = tokenizer.batch_decode(
model_output,
skip_special_tokens=True,
)
print(batch_detokenized)
đ Documentation
Model Info
| Property |
Details |
| Source Language |
Arabic |
| Target Language |
English |
| Dataset |
OPUS and HPLT data |
| Model Architecture |
Transformer-base |
| Tokenizer |
SentencePiece (Unigram) |
| Cleaning |
We used OpusCleaner with a set of basic rules. Details can be found in the filter files here. |
You can check out our deliverable report, GitHub repository, and website for more details.
Benchmarks
When decoded using Marian, the model has the following test scores.
| Test set |
BLEU |
chrF++ |
COMET22 |
| FLORES200 |
40.1 |
63.1 |
0.8645 |
| NTREX |
34.7 |
58.9 |
0.8426 |
đ License
This project is licensed under cc-by-4.0.
đ Acknowledgements
This project has received funding from the European Union's Horizon Europe research and innovation programme under grant agreement No 101070350 and from UK Research and Innovation (UKRI) under the UK government's Horizon Europe funding guarantee [grant number 10052546].
Brought to you by researchers from the University of Edinburgh and Charles University in Prague with support from the whole HPLT consortium.
%20--%3e%3cdefs%3e%3cstyle%3e%20.st0%20{%20fill:%20%23061b40;%20}%20.st1%20{%20fill:%20%23306af1;%20}%20.st2%20{%20fill:%20%235ce5cf;%20}%20%3c/style%3e%3c/defs%3e%3cg%3e%3cpath%20class='st0'%20d='M55,10.5h9v3h-9v9h12V7.5h-12v3ZM64,19.5h-6v-3h6v3Z'/%3e%3cpolygon%20class='st0'%20points='69%2016.5%2078%2016.5%2078%2019.5%2069%2019.5%2069%2022.5%2081%2022.5%2081%2013.5%2072%2013.5%2072%2010.5%2081%2010.5%2081%207.5%2069%207.5%2069%2016.5'/%3e%3cpolygon%20class='st0'%20points='95%2010.5%2095%207.5%2083%207.5%2083%2022.5%2095%2022.5%2095%2019.5%2086%2019.5%2086%2016.5%2095%2016.5%2095%2013.5%2086%2013.5%2086%2010.5%2095%2010.5'/%3e%3cpath%20class='st0'%20d='M40,1.5v21h11.6l1.4-1.4v-7.6h0c0,0-1.4-1.5-1.4-1.5l1.4-1.4V2.9l-1.4-1.4h-11.6ZM50,19.5h-7v-6h7v6ZM50,10.5h-7v-6h7v6Z'/%3e%3c/g%3e%3cpath%20class='st1'%20d='M23.1,24L14.7,4.8l-4.9,11.2h3.8l-1.8,4H3.3L12.1,0H2C.9,0,0,.9,0,2v20c0,1.1.9,2,2,2h21.1Z'/%3e%3cpath%20class='st2'%20d='M34,0h-16.8l10.6,24h6.2c1.1,0,2-.9,2-2V2C36,.9,35.1,0,34,0ZM32.5,20h-4V4h4v16Z'/%3e%3c/svg%3e)
