Aramodernbert Base V1.0

🚀 AraModernBert-base-V1.0

AraModernBert is an advanced Arabic language model. It's built on the ModernBERT architecture, significantly advancing Arabic language understanding. It combines state - of - the - art transformer design with training on 100 GiGaBytes of Arabic text.

🚀 Quick Start

Here's how to use AraModernBert with the Transformers library:

Basic Usage

from transformers import AutoTokenizer, AutoModel

# Load model and tokenizer
tokenizer = AutoTokenizer.from_pretrained("NAMAA-Space/AraModernBert-Base-V1.0")
model = AutoModel.from_pretrained("NAMAA-Space/AraModernBert-Base-V1.0")

# Encode text
text = "مرحبا بكم في عالم الذكاء الاصطناعي"
inputs = tokenizer(text, return_tensors="pt")
outputs = model(**inputs)

# Get embeddings
embeddings = outputs.last_hidden_state

Advanced Usage

from transformers import AutoTokenizer, AutoModelForMaskedLM
import torch

tokenizer = AutoTokenizer.from_pretrained("NAMAA-Space/AraModernBert-Base-V1.0")
model = AutoModelForMaskedLM.from_pretrained("NAMAA-Space/AraModernBert-Base-V1.0")

text = "الذكاء الاصطناعي هو [MASK] المستقبل."
inputs = tokenizer(text, return_tensors="pt")
token_index = torch.where(inputs["input_ids"][0] == tokenizer.mask_token_id)[0]

outputs = model(**inputs)
predictions = outputs.logits
predicted_token_id = torch.argmax(predictions[0, token_index]).item()
predicted_token = tokenizer.decode(predicted_token_id)
print(predicted_token)  

✨ Features

• Advanced Architecture: Built on the ModernBERT architecture, it combines state - of - the - art transformer design.
• Large - Scale Training: Trained on 100 GigaBytes of Arabic text.
• Custom Tokenizer: A specialized tokenizer with a vocabulary size of 50,280 tokens, optimized for Arabic language processing.
• Transtokenization Technique: Used to optimally initialize the embedding layer for MLM.
• Long Context Support: Can handle up to 8192 tokens for processing longer documents.

📚 Documentation

Model Configuration

{
  "hidden_size": 768,
  "intermediate_size": 1152,
  "num_attention_heads": 12,
  "num_hidden_layers": 22,
  "max_position_embeddings": 8192,
  "vocab_size": 50280,
  "global_attn_every_n_layers": 3,
  "local_attention": 128,
  "global_rope_theta": 160000.0,
  "local_rope_theta": 10000.0,
  "architectures": ["ModernBertForMaskedLM"],
  "model_type": "modernbert",
  "cls_token_id": 3,
  "mask_token_id": 6,
  "pad_token_id": 5,
  "sep_token_id": 4,
  "unk_token_id": 2
}

Intended Uses & Limitations

AraModernBert can be used for a wide range of Arabic NLP tasks, including:

• Text Embeddings & Representation
• Information Retrieval
• RAG (Retrieval Augmented Generation)
• Document Similarity
• Text Classification
• Sentiment Analysis

Limitations and Biases

• The model is optimized for Modern Standard Arabic and may show varying performance on dialectal Arabic variants or classical Arabic texts.
• Performance may vary across domains and specialized terminology.
• Users should be aware of potential biases present in the training data.

Evaluation Results

1. Semantic Textual Similarity (STS)

We fine - tuned the model on STS datasets to enhance semantic understanding capabilities:

• STS17: 0.831
• STS22: 0.617

Note: The STS - optimized model will be released soon as a separate checkpoint.

2. Text Classification

We finetuned AraModernBert on a multi - class classification task using the SANAD dataset.

Overall Metrics:

• AraModernBert:
  • Accuracy: 94.32%
  • F1 Score: 94.31%
  • Precision: 94.31%
  • Recall: 94.32%

Per - Class Performance (AraModernBert):

3. Named Entity Recognition (NER)

The model achieved excellent performance on Arabic NER tasks:

• Accuracy: 90.39%
• Precision: 0.7357
• Recall: 0.7442
• F1: 0.7399

Model Architecture

AraModernBert inherits the modern architecture features from ModernBERT, while adding Trans - Tokenization approach:

• 22 transformer layers with 768 hidden dimensions
• Alternating Attention mechanism with global attention every 3 layers and a local attention window of 128 tokens
• Rotary Positional Embeddings (RoPE) with different theta values for global (160000.0) and local (10000.0) attention
• 8,192 token context window for processing longer documents
• Specialized vocabulary of 50,280 tokens optimized for Arabic

Technical Specifications

Citation

If you use this model in your research, please cite:

@misc{AraModernBERT2025,
  title={AraModernBERT: Advanced Arabic Language Model Through Trans-Tokenization and ModernBERT architecture},
  author={NAMAA},
  year={2025},
  publisher={Hugging Face},
  howpublished={\url{https://huggingface.co/NAMAA-Space/AraModernBert-Base-V1.0}},
  note={Accessed: 2025-03-02}
}

Acknowledgements

This model builds upon the ModernBERT architecture developed by Answer.AI and LightOn. We acknowledge their contributions to the field of encoder - only models and extend their work to the Arabic language through our novel Trans - Tokenized approach.

@misc{modernbert,
      title={Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference}, 
      author={Benjamin Warner and Antoine Chaffin and Benjamin Clavié and Orion Weller and Oskar Hallström and Said Taghadouini and Alexis Gallagher and Raja Biswas and Faisal Ladhak and Tom Aarsen and Nathan Cooper and Griffin Adams and Jeremy Howard and Iacopo Poli},
      year={2024},
      eprint={2412.13663},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2412.13663}, 
}

@inproceedings{remy-delobelle2024transtokenization,
    title={Trans-Tokenization and Cross-lingual Vocabulary Transfers: Language Adaptation of {LLM}s for Low-Resource {NLP}},
    author={Remy, Fran{\c{c}}ois and Delobelle, Pieter and Avetisyan, Hayastan and Khabibullina, Alfiya and de Lhoneux, Miryam and Demeester, Thomas},
    booktitle={First Conference on Language Modeling},
    year={2024},
    url={https://openreview.net/forum?id=sBxvoDhvao}
}

📄 License

This model is licensed under the Apache 2.0 license.