🚀 AraBERT v1 & v2: Pre-training BERT for Arabic Language Understanding
AraBERT is an Arabic pre-trained language model based on Google's BERT architecture. It uses the same BERT-Base configuration. More details can be found in the AraBERT Paper and the AraBERT Meetup.
There are two versions of the model, AraBERTv0.1 and AraBERTv1. The difference is that AraBERTv1 uses pre-segmented text, where prefixes and suffixes are split using the Farasa Segmenter.
We evaluate AraBERT models on different downstream tasks and compare them with mBERT and other state-of-the-art models (to the best of our knowledge). The tasks include sentiment analysis on 6 different datasets (HARD, ASTD-Balanced, ArsenTD-Lev, LABR), named entity recognition with the ANERcorp, and Arabic question answering on Arabic-SQuAD and ARCD.
✨ Features
AraBERTv2
What's New!
AraBERT now comes in 4 new variants to replace the old v1 versions. More details are available in the AraBERT folder, the README, and the AraBERT Paper.
All models are available on the HuggingFace
model page under the aubmindlab name. Checkpoints are available in PyTorch, TF2, and TF1 formats.
Better Pre-Processing and New Vocab
We identified an issue with AraBERTv1's wordpiece vocabulary. The problem was that punctuations and numbers were still attached to words when learning the wordpiece vocab. Now, we insert a space between numbers and characters and around punctuation characters.
The new vocabulary was learned using the BertWordpieceTokenizer
from the tokenizers
library and should now support the Fast tokenizer implementation from the transformers
library.
P.S.: All the old BERT codes should work with the new BERT. Just change the model name and check the new preprocessing function. Please read the section on how to use the preprocessing function
Bigger Dataset and More Compute
We used approximately 3.5 times more data and trained for a longer time. For dataset sources, see the Dataset Section.
Model |
Hardware |
num of examples with seq len (128 / 512) |
128 (Batch Size/ Num of Steps) |
512 (Batch Size/ Num of Steps) |
Total Steps |
Total Time (in Days) |
AraBERTv0.2-base |
TPUv3-8 |
420M / 207M |
2560 / 1M |
384/ 2M |
3M |
- |
AraBERTv0.2-large |
TPUv3-128 |
420M / 207M |
13440 / 250K |
2056 / 300K |
550K |
7 |
AraBERTv2-base |
TPUv3-8 |
420M / 207M |
2560 / 1M |
384/ 2M |
3M |
- |
AraBERTv2-large |
TPUv3-128 |
520M / 245M |
13440 / 250K |
2056 / 300K |
550K |
7 |
AraBERT-base (v1/v0.1) |
TPUv2-8 |
- |
512 / 900K |
128 / 300K |
1.2M |
4 |
📦 Installation
It is recommended to apply our preprocessing function before training/testing on any dataset.
Install the arabert python package to segment text for AraBERT v1 & v2 or to clean your data pip install arabert
💻 Usage Examples
Basic Usage
from arabert.preprocess import ArabertPreprocessor
model_name="aubmindlab/bert-large-arabertv2"
arabert_prep = ArabertPreprocessor(model_name=model_name)
text = "ولن نبالغ إذا قلنا إن هاتف أو كمبيوتر المكتب في زمننا هذا ضروري"
arabert_prep.preprocess(text)
>>>"و+ لن نبالغ إذا قل +نا إن هاتف أو كمبيوتر ال+ مكتب في زمن +نا هذا ضروري"
📚 Documentation
Dataset
The pre-training data used for the new AraBERT model is also used for Arabic GPT2 and ELECTRA. The dataset consists of 77GB, 200,095,961 lines, 8,655,948,860 words, or 82,232,988,358 characters (before applying Farasa Segmentation).
For the new dataset, we added the unshuffled OSCAR corpus, after thorough filtering, to the previous dataset used in AraBERTv1, excluding the previously crawled websites:
Preprocessing
It is recommended to apply our preprocessing function before training/testing on any dataset.
Install the arabert python package to segment text for AraBERT v1 & v2 or to clean your data pip install arabert
TensorFlow 1.x models
The TF1.x models are available in the HuggingFace models repo. You can download them as follows:
- via git-lfs: clone all the models in a repo
curl -s https://packagecloud.io/install/repositories/github/git-lfs/script.deb.sh | sudo bash
sudo apt-get install git-lfs
git lfs install
git clone https://huggingface.co/aubmindlab/MODEL_NAME
tar -C ./MODEL_NAME -zxvf /content/MODEL_NAME/tf1_model.tar.gz
where MODEL_NAME
is any model under the aubmindlab
name.
- via
wget
:
- Go to the tf1_model.tar.gz file on huggingface.co/models/aubmindlab/MODEL_NAME.
- Copy the
oid sha256
.
- Then run
wget https://cdn-lfs.huggingface.co/aubmindlab/aragpt2-base/INSERT_THE_SHA_HERE
(e.g., for aragpt2-base
: wget https://cdn-lfs.huggingface.co/aubmindlab/aragpt2-base/3766fc03d7c2593ff2fb991d275e96b81b0ecb2098b71ff315611d052ce65248
)
📄 License
If you used this model, please cite us as:
@inproceedings{antoun2020arabert,
title={AraBERT: Transformer-based Model for Arabic Language Understanding},
author={Antoun, Wissam and Baly, Fady and Hajj, Hazem},
booktitle={LREC 2020 Workshop Language Resources and Evaluation Conference 11--16 May 2020},
pages={9}
}
Acknowledgments
Thanks to the TensorFlow Research Cloud (TFRC) for free access to Cloud TPUs. We couldn't have done it without this program. Thanks also to the AUB MIND Lab members for their continuous support. Additionally, we thank Yakshof and Assafir for providing data and storage access. Special thanks to Habib Rahal (https://www.behance.net/rahalhabib) for giving AraBERT a visual identity.
Contacts
Wissam Antoun: Linkedin | Twitter | Github | wfa07@mail.aub.edu | wissam.antoun@gmail.com
Fady Baly: Linkedin | Twitter | Github | fgb06@mail.aub.edu | baly.fady@gmail.com