Bert Base Arabert

🚀 AraBERT v1 & v2: Pre-training BERT for Arabic Language Understanding

AraBERT is an Arabic pretrained language model based on Google's BERT architecture, aiming to enhance Arabic language understanding.

📦 Dataset

The pretraining data used for the new AraBERT model is also utilized for Arabic GPT2 and ELECTRA. The dataset comprises 77GB, 200,095,961 lines, 8,655,948,860 words, or 82,232,988,358 characters (before applying Farasa Segmentation).

For the new dataset, the unshuffled OSCAR corpus, after thorough filtering, was added to the previous dataset used in AraBERTv1, excluding the previously crawled websites:

• OSCAR unshuffled and filtered.
• Arabic Wikipedia dump from 2020/09/01
• The 1.5B words Arabic Corpus
• The OSIAN Corpus
• Assafir news articles. A huge thank you to Assafir for providing the data.

🚀 Quick Start

There are two versions of the model, AraBERTv0.1 and AraBERTv1. The difference is that AraBERTv1 uses pre-segmented text where prefixes and suffixes were split using the Farasa Segmenter.

We evaluate AraBERT models on different downstream tasks and compare them to mBERT and other state-of-the-art models (To the extent of our knowledge). The tasks include Sentiment Analysis on 6 different datasets (HARD, ASTD-Balanced, ArsenTD-Lev, LABR), Named Entity Recognition with the ANERcorp, and Arabic Question Answering on Arabic-SQuAD and ARCD.

💻 Usage Examples

Basic Usage

from arabert.preprocess import ArabertPreprocessor

model_name="bert-base-arabert"
arabert_prep = ArabertPreprocessor(model_name=model_name)

text = "ولن نبالغ إذا قلنا إن هاتف أو كمبيوتر المكتب في زمننا هذا ضروري"
arabert_prep.preprocess(text)
>>>"و+ لن نبالغ إذا قل +نا إن هاتف أو كمبيوتر ال+ مكتب في زمن +نا هذا ضروري"

Accepted_models

bert-base-arabertv01
bert-base-arabert
bert-base-arabertv02
bert-base-arabertv2
bert-large-arabertv02
bert-large-arabertv2
araelectra-base
aragpt2-base
aragpt2-medium
aragpt2-large
aragpt2-mega

📚 Documentation

AraBERTv2

What's New!

AraBERT now comes in 4 new variants to replace the old v1 versions:

More details are available in the AraBERT folder, the README, and the AraBERT Paper.

All models are available in the HuggingFace model page under the aubmindlab name. Checkpoints are available in PyTorch, TF2, and TF1 formats.

Better Pre-Processing and New Vocab

We identified an issue with AraBERTv1's wordpiece vocabulary. The issue stemmed from punctuations and numbers that were still attached to words when the wordpiece vocab was learned. We now insert a space between numbers and characters and around punctuation characters.

The new vocabulary was learned using the BertWordpieceTokenizer from the tokenizers library and should now support the Fast tokenizer implementation from the transformers library.

P.S.: All the old BERT codes should work with the new BERT. Just change the model name and check the new preprocessing function. Please read the section on how to use the preprocessing function

Bigger Dataset and More Compute

We used ~3.5 times more data and trained for a longer period. For Dataset Sources, see the Dataset Section.

TensorFlow 1.x models

The TF1.x model is available in the HuggingFace models repo. You can download them as follows:

• via git-lfs: clone all the models in a repo

curl -s https://packagecloud.io/install/repositories/github/git-lfs/script.deb.sh | sudo bash
sudo apt-get install git-lfs
git lfs install
git clone https://huggingface.co/aubmindlab/MODEL_NAME
tar -C ./MODEL_NAME -zxvf /content/MODEL_NAME/tf1_model.tar.gz

where MODEL_NAME is any model under the aubmindlab name.

• via wget:
  • Go to the tf1_model.tar.gz file on huggingface.co/models/aubmindlab/MODEL_NAME.
  • Copy the oid sha256.
  • Then run wget https://cdn-lfs.huggingface.co/aubmindlab/aragpt2-base/INSERT_THE_SHA_HERE (e.g., for aragpt2-base: wget https://cdn-lfs.huggingface.co/aubmindlab/aragpt2-base/3766fc03d7c2593ff2fb991d275e96b81b0ecb2098b71ff315611d052ce65248).

🔧 Technical Details

If you used this model, please cite us as: Google Scholar has our Bibtex wrong (missing name), use this instead

@inproceedings{antoun2020arabert,
  title={AraBERT: Transformer-based Model for Arabic Language Understanding},
  author={Antoun, Wissam and Baly, Fady and Hajj, Hazem},
  booktitle={LREC 2020 Workshop Language Resources and Evaluation Conference 11--16 May 2020},
  pages={9}
}

📄 License

Thanks to TensorFlow Research Cloud (TFRC) for the free access to Cloud TPUs. We couldn't have done it without this program. Thanks also to the AUB MIND Lab Members for their continuous support. Additionally, thanks to Yakshof and Assafir for data and storage access. Another thanks to Habib Rahal (https://www.behance.net/rahalhabib) for putting a face to AraBERT.

Contacts

Wissam Antoun: Linkedin | Twitter | Github | wfa07@mail.aub.edu | wissam.antoun@gmail.com

Fady Baly: Linkedin | Twitter | Github | fgb06@mail.aub.edu | baly.fady@gmail.com

⚠️ Important Note

A newer version of this model is available: AraBERTv2