Mbart Large 50 Finetuned Opus Pt En Translation

🚀 mBART-large-50 for Portuguese to English Translation

This is an mBART-large-50 model fine-tuned on opus100 and opusbook for Portuguese to English translation. It's based on the mBART-50 architecture and fine-tuned on the opus100 dataset for the NMT downstream task.

✨ Features

• Multilingual Support: mBART-50 supports 50 languages, enabling multilingual machine translation.
• Fine-tuned for NMT: Fine-tuned on the opus100 and opusbook datasets for Portuguese to English translation.
• Good Performance: Achieved a BLEU score of 26.12 on the test set.

📦 Installation

git clone https://github.com/huggingface/transformers.git
pip install -q ./transformers

💻 Usage Examples

Basic Usage

from transformers import MBart50TokenizerFast, MBartForConditionalGeneration

ckpt = 'Narrativa/mbart-large-50-finetuned-opus-pt-en-translation'

tokenizer = MBart50TokenizerFast.from_pretrained(ckpt)
model = MBartForConditionalGeneration.from_pretrained(ckpt).to("cuda")

tokenizer.src_lang = 'pt_XX'

def translate(text):
    inputs = tokenizer(text, return_tensors='pt')
    input_ids = inputs.input_ids.to('cuda')
    attention_mask = inputs.attention_mask.to('cuda')
    output = model.generate(input_ids, attention_mask=attention_mask, forced_bos_token_id=tokenizer.lang_code_to_id['en_XX'])
    return tokenizer.decode(output[0], skip_special_tokens=True)
    
    
translate('here your Portuguese text to be translated to English...')

📚 Documentation

Details of mBART-50 🧠

mBART-50 is a multilingual Sequence-to-Sequence model pre-trained using the "Multilingual Denoising Pretraining" objective. It was introduced in Multilingual Translation with Extensible Multilingual Pretraining and Finetuning paper.

mBART-50 is a multilingual Sequence-to-Sequence model. It was created to show that multilingual translation models can be created through multilingual fine-tuning. Instead of fine-tuning on one direction, a pre-trained model is fine-tuned many directions simultaneously. mBART-50 is created using the original mBART model and extended to add extra 25 languages to support multilingual machine translation models of 50 languages. The pre-training objective is explained below.

Multilingual Denoising Pretraining: The model incorporates N languages by concatenating data: D = {D1, ..., DN } where each Di is a collection of monolingual documents in language i. The source documents are noised using two schemes, first randomly shuffling the original sentences' order, and second a novel in-filling scheme, where spans of text are replaced with a single mask token. The model is then tasked to reconstruct the original text. 35% of each instance's words are masked by random sampling a span length according to a Poisson distribution (λ = 3.5). The decoder input is the original text with one position offset. A language id symbol LID is used as the initial token to predict the sentence.

Details of the downstream task (NMT) - Dataset 📚

• Homepage: Link
• Repository: GitHub
• Paper: ARXIV

Dataset Summary

OPUS-100 is English-centric, meaning that all training pairs include English on either the source or target side. The corpus covers 100 languages (including English). Languages were selected based on the volume of parallel data available in OPUS.

Languages

OPUS-100 contains approximately 55M sentence pairs. Of the 99 language pairs, 44 have 1M sentence pairs of training data, 73 have at least 100k, and 95 have at least 10k.

Dataset Structure

Data Fields

• src_tag: string text in source language
• tgt_tag: string translation of source language in target language

Data Splits

The dataset is split into training, development, and test portions. Data was prepared by randomly sampled up to 1M sentence pairs per language pair for training and up to 2000 each for development and test. To ensure that there was no overlap (at the monolingual sentence level) between the training and development/test data, they applied a filter during sampling to exclude sentences that had already been sampled. Note that this was done cross-lingually so that, for instance, an English sentence in the Portuguese-English portion of the training data could not occur in the Hindi-English test set.

Test set metrics 🧾

We got a BLEU score of 26.12

🔧 Technical Details

The pre-training of mBART-50 uses the Multilingual Denoising Pretraining method. The model incorporates multiple languages by concatenating data. The source documents are noised using two schemes, and then the model is tasked to reconstruct the original text. The decoder input is the original text with one position offset, and a language id symbol LID is used as the initial token to predict the sentence.

📄 License

Not provided in the original README.

Created by: Narrativa

About Narrativa: Natural Language Generation (NLG) | Gabriele, our machine learning-based platform, builds and deploys natural language solutions. #NLG #AI