Albert Large Arabic

🚀 Arabic-ALBERT Large

An Arabic edition of the ALBERT Large pretrained language model, which offers advanced language understanding capabilities for Arabic text.

If you use any of these models in your work, please cite this work as:

@software{ali_safaya_2020_4718724,
  author       = {Ali Safaya},
  title        = {Arabic-ALBERT},
  month        = aug,
  year         = 2020,
  publisher    = {Zenodo},
  version      = {1.0.0},
  doi          = {10.5281/zenodo.4718724},
  url          = {https://doi.org/10.5281/zenodo.4718724}
}

🚀 Quick Start

You can use these models by installing torch or tensorflow and Huggingface library transformers. And you can use it directly by initializing it like this:

from transformers import AutoTokenizer, AutoModel

# loading the tokenizer
tokenizer    = AutoTokenizer.from_pretrained("kuisailab/albert-large-arabic")

# loading the model
model   = AutoModelForMaskedLM.from_pretrained("kuisailab/albert-large-arabic")

✨ Features

• Multilingual Adaptability: The model is trained on a diverse dataset including the Arabic version of OSCAR and Arabic Wikipedia, making it suitable for various Arabic language tasks.
• Customized Training: The pretraining procedure follows the training settings of BERT with some optimizations, such as 7M training steps with a batch size of 64.

📦 Installation

To use the Arabic - ALBERT models, you need to install the following libraries:

• torch or tensorflow
• Huggingface library transformers

📚 Documentation

Pretraining data

The models were pretrained on ~4.4 Billion words:

• Arabic version of OSCAR (unshuffled version of the corpus) - filtered from Common Crawl
• Recent dump of Arabic Wikipedia

Notes on training data:

• Our final version of the corpus contains some non - Arabic words inlines, which we did not remove from sentences since that would affect some tasks like NER.
• Although non - Arabic characters were lowered as a preprocessing step, since Arabic characters do not have upper or lower case, there is no cased and uncased version of the model.
• The corpus and vocabulary set are not restricted to Modern Standard Arabic, they contain some dialectical Arabic too.

Pretraining details

• These models were trained using Google ALBERT's github repository on a single TPU v3 - 8 provided for free from TFRC.
• Our pretraining procedure follows training settings of BERT with some changes: trained for 7M training steps with a batch size of 64, instead of 125K with a batch size of 4096.

Models

Results

For further details on the models' performance or any other queries, please refer to [Arabic - ALBERT](https://github.com/KUIS - AI - Lab/Arabic - ALBERT/)

🔧 Technical Details

• Training Environment: The models were trained on a single TPU v3 - 8 provided by TFRC using Google ALBERT's github repository.
• Training Procedure: The training settings are based on BERT with adjustments. It was trained for 7M training steps with a batch size of 64, different from the original BERT's 125K steps with a batch size of 4096.

📄 License

No license information is provided in the original document.

💡 Usage Tip

Thanks to Google for providing free TPU for the training process and for Huggingface for hosting these models on their servers 😊