Bert Base Arabertv01

🚀 AraBERT v1 & v2: Pre - training BERT for Arabic Language Understanding

AraBERT is an Arabic pre - trained language model based on Google's BERT architecture. It uses the same BERT - Base config. More details can be found in the AraBERT Paper and in the AraBERT Meetup. This model aims to enhance Arabic language understanding, offering high - quality pre - training for various downstream tasks.

📋 Dataset Information

⚠️ Important Note

A newer version of this model is available: AraBERTv02

✨ Features

• Two Model Versions: There are two versions, AraBERTv0.1 and AraBERTv1. AraBERTv1 uses pre - segmented text where prefixes and suffixes are split using the Farasa Segmenter.
• Downstream Task Evaluation: Evaluated on different downstream tasks such as Sentiment Analysis on 6 different datasets ([HARD](https://github.com/elnagara/HARD - Arabic - Dataset), [ASTD - Balanced](https://www.aclweb.org/anthology/D15 - 1299), [ArsenTD - Lev](https://staff.aub.edu.lb/~we07/Publications/ArSentD - LEV_Sentiment_Corpus.pdf), LABR), Named Entity Recognition with the ANERcorp, and Arabic Question Answering on Arabic - SQuAD and ARCD.

📦 Installation

Preprocessing Installation

It is recommended to apply the preprocessing function before training/testing on any dataset. Install farasapy to segment text for AraBERT v1 & v2:

pip install farasapy

TensorFlow 1.x Models Installation

The TF1.x models are available in the HuggingFace models repo. You can download them in the following ways:

• Via git - lfs: Clone all the models in a repo

curl -s https://packagecloud.io/install/repositories/github/git - lfs/script.deb.sh | sudo bash
sudo apt - get install git - lfs
git lfs install
git clone https://huggingface.co/aubmindlab/MODEL_NAME
tar -C ./MODEL_NAME -zxvf /content/MODEL_NAME/tf1_model.tar.gz

where MODEL_NAME is any model under the aubmindlab name.

• Via wget:
  • Go to the tf1_model.tar.gz file on huggingface.co/models/aubmindlab/MODEL_NAME.
  • Copy the oid sha256.
  • Then run wget https://cdn - lfs.huggingface.co/aubmindlab/aragpt2 - base/INSERT_THE_SHA_HERE (e.g., for aragpt2 - base: wget https://cdn - lfs.huggingface.co/aubmindlab/aragpt2 - base/3766fc03d7c2593ff2fb991d275e96b81b0ecb2098b71ff315611d052ce65248).

💻 Usage Examples

Preprocessing Example

from arabert.preprocess import ArabertPreprocessor

model_name="bert-base-arabertv01"
arabert_prep = ArabertPreprocessor(model_name=model_name)

text = "ولن نبالغ إذا قلنا إن هاتف أو كمبيوتر المكتب في زمننا هذا ضروري"
arabert_prep.preprocess(text)

Accepted Models

bert-base-arabertv01
bert-base-arabert
bert-base-arabertv02
bert-base-arabertv2
bert-large-arabertv02
bert-large-arabertv2
araelectra-base
aragpt2-base
aragpt2-medium
aragpt2-large
aragpt2-mega

📚 Documentation

AraBERTv2

What's New!

AraBERT now has 4 new variants to replace the old v1 versions. More details are available in the AraBERT folder, the [README](https://github.com/aub - mind/arabert/blob/master/AraBERT/README.md), and the AraBERT Paper.

All models are available in the HuggingFace model page under the aubmindlab name. Checkpoints are available in PyTorch, TF2, and TF1 formats.

Better Pre - Processing and New Vocab

An issue with AraBERTv1's wordpiece vocabulary was identified. The problem came from punctuations and numbers that were still attached to words when the wordpiece vocab was learned. Now, a space is inserted between numbers and characters and around punctuation characters.

The new vocabulary was learned using the BertWordpieceTokenizer from the tokenizers library and should support the Fast tokenizer implementation from the transformers library.

P.S.: All the old BERT codes should work with the new BERT. Just change the model name and check the new preprocessing function. Please read the section on how to use the preprocessing function

Bigger Dataset and More Compute

Approximately 3.5 times more data was used, and the models were trained for a longer time. For dataset sources, see the Dataset Section.

Dataset

The pretraining data used for the new AraBERT model is also used for Arabic GPT2 and ELECTRA. The dataset consists of 77GB or 200,095,961 lines or 8,655,948,860 words or 82,232,988,358 chars (before applying Farasa Segmentation).

For the new dataset, the unshuffled OSCAR corpus was added after thorough filtering to the previous dataset used in AraBERTv1, excluding the previously crawled websites:

• OSCAR unshuffled and filtered.
• [Arabic Wikipedia dump](https://archive.org/details/arwiki - 20190201) from 2020/09/01
• [The 1.5B words Arabic Corpus](https://www.semanticscholar.org/paper/1.5 - billion - words - Arabic - Corpus - El - Khair/f3eeef4afb81223df96575adadf808fe7fe440b4)
• [The OSIAN Corpus](https://www.aclweb.org/anthology/W19 - 4619)
• Assafir news articles. A huge thank you to Assafir for providing the data.

Preprocessing

It is recommended to apply the preprocessing function before training/testing on any dataset. Install farasapy to segment text for AraBERT v1 & v2 pip install farasapy

🔧 Technical Details

The model is based on [Google's BERT architecture](https://github.com/google - research/bert) and uses the BERT - Base config. Different versions of the model have different pre - processing and training setups. For example, AraBERTv1 uses pre - segmented text, and the new AraBERTv2 models have improvements in pre - processing, vocabulary, and use a larger dataset.

📄 License

No license information was provided in the original document.

📖 Citation

If you used this model, please cite us as:

@inproceedings{antoun2020arabert,
  title={AraBERT: Transformer - based Model for Arabic Language Understanding},
  author={Antoun, Wissam and Baly, Fady and Hajj, Hazem},
  booktitle={LREC 2020 Workshop Language Resources and Evaluation Conference 11--16 May 2020},
  pages={9}
}

🙏 Acknowledgments

Thanks to the TensorFlow Research Cloud (TFRC) for free access to Cloud TPUs. Without this program, this work could not have been completed. Thanks also to the AUB MIND Lab members for continuous support. Additionally, thanks to Yakshof and Assafir for data and storage access. Another thanks to Habib Rahal (https://www.behance.net/rahalhabib) for giving a face to AraBERT.

📞 Contacts

• Wissam Antoun: [Linkedin](https://www.linkedin.com/in/wissam - antoun - 622142b4/) | Twitter | Github | wfa07@mail.aub.edu | wissam.antoun@gmail.com
• Fady Baly: Linkedin | Twitter | Github | fgb06@mail.aub.edu | baly.fady@gmail.com