Bert Large Arabic

🚀 Arabic BERT Large Model

A pretrained BERT Large language model tailored for Arabic, offering advanced language understanding capabilities for Arabic text.

🚀 Quick Start

You can use this model by installing torch or tensorflow and Huggingface library transformers. And you can use it directly by initializing it like this:

from transformers import AutoTokenizer, AutoModel

tokenizer = AutoTokenizer.from_pretrained("asafaya/bert-large-arabic")
model = AutoModelForMaskedLM.from_pretrained("asafaya/bert-large-arabic")

✨ Features

• Pretrained on a vast corpus of ~8.2 Billion words, including the Arabic version of OSCAR and a recent dump of Arabic Wikipedia, along with other Arabic resources totaling ~95GB of text.
• The model's corpus contains some non - Arabic words inlines, which are kept to avoid affecting tasks like NER.
• There is no cased and uncased version of the model as Arabic characters do not have upper or lower case.
• The corpus and vocabulary set cover both Modern Standard Arabic and some dialectical Arabic.

📦 Installation

To use this model, you need to install torch or tensorflow and the Huggingface library transformers.

📚 Documentation

Pretraining Corpus

The arabic - bert - large model was pretrained on ~8.2 Billion words:

• Arabic version of OSCAR - filtered from Common Crawl
• Recent dump of Arabic Wikipedia

and other Arabic resources which sum up to ~95GB of text.

Notes on training data:

• Our final version of corpus contains some non - Arabic words inlines, which we did not remove from sentences since that would affect some tasks like NER.
• Although non - Arabic characters were lowered as a preprocessing step, since Arabic characters does not have upper or lower case, there is no cased and uncased version of the model.
• The corpus and vocabulary set are not restricted to Modern Standard Arabic, they contain some dialectical Arabic too.

Pretraining details

• This model was trained using Google BERT's github repository on a single TPU v3 - 8 provided for free from TFRC.
• Our pretraining procedure follows training settings of bert with some changes: trained for 3M training steps with batchsize of 128, instead of 1M with batchsize of 256.

Results

For further details on the models performance or any other queries, please refer to [Arabic - BERT](https://github.com/alisafaya/Arabic - BERT)

📄 License

If you use this model in your work, please cite this paper:

@inproceedings{safaya-etal-2020-kuisail,
    title = "{KUISAIL} at {S}em{E}val-2020 Task 12: {BERT}-{CNN} for Offensive Speech Identification in Social Media",
    author = "Safaya, Ali  and
      Abdullatif, Moutasem  and
      Yuret, Deniz",
    booktitle = "Proceedings of the Fourteenth Workshop on Semantic Evaluation",
    month = dec,
    year = "2020",
    address = "Barcelona (online)",
    publisher = "International Committee for Computational Linguistics",
    url = "https://www.aclweb.org/anthology/2020.semeval-1.271",
    pages = "2054--2059",
}

🔧 Technical Details

• Model Type: Pretrained BERT Large language model for Arabic
• Training Data:
  • Arabic version of OSCAR filtered from Common Crawl
  • Recent dump of Arabic Wikipedia
  • Other Arabic resources totaling ~95GB of text

💡 Usage Tip

Thanks to Google for providing free TPU for the training process and for Huggingface for hosting this model on their servers 😊