Bert Medium Arabic

🚀 Arabic BERT Medium Model

A pretrained BERT Medium language model tailored for Arabic, offering robust language understanding capabilities.

If you utilize this model in your work, kindly cite the following paper:

@inproceedings{safaya-etal-2020-kuisail,
    title = "{KUISAIL} at {S}em{E}val-2020 Task 12: {BERT}-{CNN} for Offensive Speech Identification in Social Media",
    author = "Safaya, Ali  and
      Abdullatif, Moutasem  and
      Yuret, Deniz",
    booktitle = "Proceedings of the Fourteenth Workshop on Semantic Evaluation",
    month = dec,
    year = "2020",
    address = "Barcelona (online)",
    publisher = "International Committee for Computational Linguistics",
    url = "https://www.aclweb.org/anthology/2020.semeval-1.271",
    pages = "2054--2059",
}

📚 Documentation

Pretraining Corpus

The arabic-bert-medium model underwent pretraining on approximately 8.2 billion words, sourced from:

• The Arabic version of OSCAR, filtered from Common Crawl
• A recent dump of Arabic Wikipedia

alongside other Arabic resources, amounting to around 95GB of text.

Notes on training data:

• Our final corpus version includes some non-Arabic words within sentences. We refrained from removing them as it could impact tasks like NER.
• Although non-Arabic characters were lowercased during preprocessing, since Arabic characters lack upper and lower cases, there are no cased and uncased versions of the model.
• The corpus and vocabulary set are not limited to Modern Standard Arabic; they also contain some dialectical Arabic.

Pretraining details

• This model was trained using Google BERT's GitHub repository on a single TPU v3 - 8, provided free of charge by TFRC.
• Our pretraining procedure adheres to BERT's training settings with some modifications: it was trained for 3M training steps with a batch size of 128, instead of 1M with a batch size of 256.

💻 Usage Examples

Basic Usage

from transformers import AutoTokenizer, AutoModel

tokenizer = AutoTokenizer.from_pretrained("asafaya/bert-medium-arabic")
model = AutoModelForMaskedLM.from_pretrained("asafaya/bert-medium-arabic")

📊 Results

For more detailed information on the model's performance or any other inquiries, please refer to Arabic - BERT.

🙏 Acknowledgement

We express our gratitude to Google for providing free TPU for the training process and to Huggingface for hosting this model on their servers 😊