Xlm Roberta Large Qe V1

🚀 Quality Estimation for Machine Translation

This model offers a reference - free quality estimation (QE) solution for machine translation (MT) systems. It's a fine - tuned version of [FacebookAI/xlm - roberta - large](https://huggingface.co/FacebookAI/xlm - roberta - large) on the ymoslem/wmt - da - human - evaluation dataset, delivering valuable insights into MT quality.

🚀 Quick Start

Installation

1. Install the required libraries.

pip3 install --upgrade datasets accelerate transformers
pip3 install --upgrade flash_attn triton

Inference Steps

2. Load the test dataset.

from datasets import load_dataset

test_dataset = load_dataset("ymoslem/wmt-da-human-evaluation",
                             split="test",
                             trust_remote_code=True
                            )
print(test_dataset)

3. Load the model and tokenizer:

from transformers import AutoModelForSequenceClassification, AutoTokenizer
import torch

# Load the fine-tuned model and tokenizer
model_name = "ymoslem/ModernBERT-large-qe-v1"
model = AutoModelForSequenceClassification.from_pretrained(
    model_name,
    device_map="auto",
    torch_dtype=torch.bfloat16,
    attn_implementation="flash_attention_2",
)
tokenizer = AutoTokenizer.from_pretrained(model_name)

# Move model to GPU if available
device = "cuda" if torch.cuda.is_available() else "cpu"
model.to(device)
model.eval()

4. Prepare the dataset. Each source segment src and target segment tgt are separated by the sep_token, which is '</s>' for ModernBERT.

sep_token = tokenizer.sep_token
input_test_texts = [f"{src} {sep_token} {tgt}" for src, tgt in zip(test_dataset["src"], test_dataset["mt"])]

5. Generate predictions.

Basic Usage

If you print model.config.problem_type, the output is regression. Still, you can use the "text - classification" pipeline as follows (cf. pipeline documentation):

from transformers import pipeline

classifier = pipeline("text-classification",
                      model=model_name,
                      tokenizer=tokenizer,
                      device=0,
                     )

predictions = classifier(input_test_texts,
                         batch_size=128,
                         truncation=True,
                         padding="max_length",
                         max_length=tokenizer.model_max_length,
                       )
predictions = [prediction["score"] for prediction in predictions]

Advanced Usage

Alternatively, you can use an elaborate version of the code, which is slightly faster and provides more control.

from torch.utils.data import DataLoader
import torch
from tqdm.auto import tqdm

# Tokenization function
def process_batch(batch, tokenizer, device):
    sep_token = tokenizer.sep_token
    input_texts = [f"{src} {sep_token} {tgt}" for src, tgt in zip(batch["src"], batch["mt"])]
    tokens = tokenizer(input_texts,
                       truncation=True,
                       padding="max_length",
                       max_length=tokenizer.model_max_length,
                       return_tensors="pt",
                      ).to(device)
    return tokens
    


# Create a DataLoader for batching
test_dataloader = DataLoader(test_dataset, 
                             batch_size=128,   # Adjust batch size as needed
                             shuffle=False)


# List to store all predictions
predictions = []

with torch.no_grad():
    for batch in tqdm(test_dataloader, desc="Inference Progress", unit="batch"):

        tokens = process_batch(batch, tokenizer, device)
        
        # Forward pass: Generate model's logits
        outputs = model(**tokens)

        # Get logits (predictions)
        logits = outputs.logits

        # Extract the regression predicted values
        batch_predictions = logits.squeeze()

        # Extend the list with the predictions
        predictions.extend(batch_predictions.tolist())

✨ Features

• Multilingual Support: Supports multiple languages including bn, cs, de, en, et, fi, fr, gu, ha, hi, is, ja, kk, km, lt, lv, pl, ps, ru, ta, tr, uk, xh, zh, zu.
• Reference - Free QE: Enables quality estimation of machine translation systems without the need for reference translations.

📚 Documentation

Model description

This model is for reference - free quality estimation (QE) of machine translation (MT) systems.

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 8e - 05
• train_batch_size: 64
• eval_batch_size: 64
• seed: 42
• optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.999) and epsilon = 1e - 08 and optimizer_args = No additional optimizer arguments
• lr_scheduler_type: linear
• training_steps: 20000

Training results

Framework versions

• Transformers 4.48.0
• Pytorch 2.4.1+cu124
• Datasets 3.2.0
• Tokenizers 0.21.0

🔧 Technical Details

This model is a fine - tuned version of [FacebookAI/xlm - roberta - large](https://huggingface.co/FacebookAI/xlm - roberta - large). It achieves the following results on the evaluation set:

• Loss: 0.0752
• Metrics:
  • Pearson Correlation: 0.422
  • Mean Absolute Error: 0.196
  • Root Mean Squared Error: 0.245
  • R - Squared: 0.245

📄 License

This project is licensed under the MIT license.

📦 Model Information