EXLMR

🚀 XLM-R and EXLMR Model

This project introduces the EXLMR model, an extension of XLM - R. It expands the tokenizer vocabulary to address out - of - vocabulary issues and benefits both low - and high - resource languages.

🚀 Quick Start

The EXLMR model is an effective solution for NLP tasks, especially for languages with limited resources. You can use this model with the transformers library for various NLP tasks.

from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch
# Define the model checkpoint
checkpoint = "Hailay/EXLMR"  

# Load the tokenizer and model
tokenizer = AutoTokenizer.from_pretrained(checkpoint)
model = AutoModelForSequenceClassification.from_pretrained(checkpoint)

✨ Features

• Extended Vocabulary: The EXLMR model expands the tokenizer vocabulary of XLM - R to incorporate new languages and alleviate out - of - vocabulary (OOV) issues.
• Multilingual Support: It supports multiple languages, including those spoken in Ethiopia such as Amharic, Tigrinya, and Afaan Oromo, and can be fine - tuned for cross - lingual tasks.
• Performance Improvement: Our approach benefits both low - resource and high - resource languages, enhancing the model's performance.

📚 Documentation

Model Overview

The XLM - R (Cross - lingual Language Model - RoBERTa) is a multilingual model trained on 100 languages. The EXLMR (Extended XLM - RoBERTa) is an extended version designed to improve performance on low - resource languages spoken in Ethiopia, including Amharic, Tigrinya, and Afaan Oromo.

Model Details

Usage

EXLMR addresses tokenization issues inherent to the XLM - R model, such as out - of - vocabulary (OOV) tokens and over - tokenization, especially for low - resource languages. Fine - tuning on specific datasets will help adapt the model to particular tasks and improve its performance.

EXLMR has been designed to support underrepresented languages, particularly those spoken in Ethiopia. Like XLM - RoBERTa, EXLMR can be finetuned to handle multiple languages simultaneously, making it effective for cross - lingual tasks such as machine translation, multilingual text classification, and question answering. EXLMR - base follows the same architecture as RoBERTa - base, with 12 layers, 768 hidden dimensions, and 12 attention heads, totaling approximately 270M parameters.

📄 License

This project is licensed under the Apache - 2.0 license.