Translate Ar En V1.0 Hplt

🚀 HPLT MT release v1.0

This repository offers a translation model for Arabic-English, trained solely with HPLT data. It's available in both Marian and Hugging Face formats.

🚀 Quick Start

This repository contains a translation model for Arabic-English trained exclusively with HPLT data. It comes in both Marian and Hugging Face formats, providing flexibility for different usage scenarios.

✨ Features

• Dual Format Availability: The model is available in both Marian and Hugging Face formats, catering to a wider range of users.
• Trained on HPLT Data: Utilizes HPLT data, ensuring high - quality training for Arabic - English translation.

📦 Installation

This section does not have specific installation steps, so it is skipped.

💻 Usage Examples

Basic Usage

Note that for quality considerations, we recommend using HPLT/translate-ar-en-v1.0-hplt_opus instead of this model.

The model has been trained with MarianNMT and the weights are in the Marian format. We have also converted the model into the Hugging Face format so it is compatible with transformers.

Using Marian

To run inference with MarianNMT, refer to the Inference/Decoding/Translation section of our GitHub repository. You will need the model file model.npz.best-chrf.npz and the vocabulary file model.ar-en.spm from this repository.

Using transformers

We have also converted this model to the Hugging Face format and you can get started with the script below. Note that due a known issue in weight conversion, the checkpoint cannot work with transformer versions <4.26 or >4.30. We tested and suggest pip install transformers==4.28.

from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

tokenizer = AutoTokenizer.from_pretrained("HPLT/translate-ar-en-v1.0-hplt")
model = AutoModelForSeq2SeqLM.from_pretrained("HPLT/translate-ar-en-v1.0-hplt")

inputs = ["Input goes here.", "Make sure the language is right."]
batch_tokenized = tokenizer(inputs, return_tensors="pt", padding=True)
model_output = model.generate(
    **batch_tokenized, num_beams=6, max_new_tokens=512
)
batch_detokenized = tokenizer.batch_decode(
    model_output,
    skip_special_tokens=True,
)

print(batch_detokenized)

📚 Documentation

Model Info

You can check out our deliverable report, GitHub repository, and website for more details.

Benchmarks

When decoded using Marian, the model has the following test scores.

Acknowledgements

This project has received funding from the European Union's Horizon Europe research and innovation programme under grant agreement No 101070350 and from UK Research and Innovation (UKRI) under the UK government's Horizon Europe funding guarantee [grant number 10052546].

Brought to you by researchers from the University of Edinburgh and Charles University in Prague with support from the whole HPLT consortium.

📄 License

This project is licensed under the CC - BY - 4.0 license.