Bert Base Arabic Camelbert Msa Sixteenth

🚀 CAMeLBERT: A collection of pre-trained models for Arabic NLP tasks

CAMeLBERT is a collection of BERT models pre-trained on Arabic texts. It offers models of different sizes and variants, including those for Modern Standard Arabic (MSA), dialectal Arabic (DA), classical Arabic (CA), and a mixed model. Additionally, there are models pre - trained on scaled - down MSA datasets. This model card focuses on CAMeLBERT-MSA-sixteenth (bert-base-arabic-camelbert-msa-sixteenth), pre - trained on a sixteenth of the full MSA dataset.

🚀 Quick Start

You can use the released model for masked language modeling or next sentence prediction. It is mainly designed to be fine - tuned on NLP tasks like NER, POS tagging, sentiment analysis, dialect identification, and poetry classification. The fine - tuning code is available here.

Usage Examples

Basic Usage

>>> from transformers import pipeline
>>> unmasker = pipeline('fill-mask', model='CAMeL-Lab/bert-base-arabic-camelbert-msa-sixteenth')
>>> unmasker("الهدف من الحياة هو [MASK] .")
[{'sequence': '[CLS] الهدف من الحياة هو التغيير. [SEP]',
  'score': 0.08320745080709457,
  'token': 7946,
  'token_str': 'التغيير'},
 {'sequence': '[CLS] الهدف من الحياة هو التعلم. [SEP]',
  'score': 0.04305094853043556,
  'token': 12554,
  'token_str': 'التعلم'},
 {'sequence': '[CLS] الهدف من الحياة هو العمل. [SEP]',
  'score': 0.0417640283703804,
  'token': 2854,
  'token_str': 'العمل'},
 {'sequence': '[CLS] الهدف من الحياة هو الحياة. [SEP]',
  'score': 0.041371218860149384,
  'token': 3696,
  'token_str': 'الحياة'},
 {'sequence': '[CLS] الهدف من الحياة هو المعرفة. [SEP]',
  'score': 0.039794355630874634,
  'token': 7344,
  'token_str': 'المعرفة'}]

Note: to download our models, you need transformers>=3.5.0. Otherwise, you can download the models manually.

Advanced Usage

Here is how to use this model to get the features of a given text in PyTorch:

from transformers import AutoTokenizer, AutoModel
tokenizer = AutoTokenizer.from_pretrained('CAMeL-Lab/bert-base-arabic-camelbert-msa-sixteenth')
model = AutoModel.from_pretrained('CAMeL-Lab/bert-base-arabic-camelbert-msa-sixteenth')
text = "مرحبا يا عالم."
encoded_input = tokenizer(text, return_tensors='pt')
output = model(**encoded_input)

And in TensorFlow:

from transformers import AutoTokenizer, TFAutoModel
tokenizer = AutoTokenizer.from_pretrained('CAMeL-Lab/bert-base-arabic-camelbert-msa-sixteenth')
model = TFAutoModel.from_pretrained('CAMeL-Lab/bert-base-arabic-camelbert-msa-sixteenth')
text = "مرحبا يا عالم."
encoded_input = tokenizer(text, return_tensors='tf')
output = model(encoded_input)

✨ Features

• Multiple Variants: Offers models for different Arabic variants (MSA, DA, CA) and a mixed model.
• Scaled - down Models: Provides models pre - trained on scaled - down MSA datasets.

📦 Installation

To use the models, you need to have transformers>=3.5.0 installed. You can install it using the following command:

pip install transformers>=3.5.0

📚 Documentation

Model description

CAMeLBERT consists of BERT models pre - trained on Arabic texts. The details are described in the paper "The Interplay of Variant, Size, and Task Type in Arabic Pre - trained Language Models."

Intended uses

The model can be used for masked language modeling, next sentence prediction, and fine - tuned on various NLP tasks.

Training data

• MSA (Modern Standard Arabic)
  • The Arabic Gigaword Fifth Edition
  • Abu El - Khair Corpus
  • OSIAN corpus
  • Arabic Wikipedia
  • The unshuffled version of the Arabic [OSCAR corpus](https://oscar - corpus.com/)

Training procedure

We use [the original implementation](https://github.com/google - research/bert) released by Google for pre - training and follow the original English BERT model's hyperparameters, unless otherwise specified.

Preprocessing

• After extracting raw text from each corpus, we perform the following pre - processing:
  • Remove invalid characters and normalize white spaces using utilities from [the original BERT implementation](https://github.com/google - research/bert/blob/eedf5716ce1268e56f0a50264a88cafad334ac61/tokenization.py#L286 - L297).
  • Remove lines without Arabic characters.
  • Remove diacritics and kashida using [CAMeL Tools](https://github.com/CAMeL - Lab/camel_tools).
  • Split each line into sentences using a heuristics - based sentence segmenter.
  • Train a WordPiece tokenizer on the entire dataset (167 GB text) with a vocabulary size of 30,000 using HuggingFace's tokenizers.
  • Do not lowercase letters nor strip accents.

Pre - training

• The model was trained on a single cloud TPU (v3 - 8) for one million steps.
• The first 90,000 steps were trained with a batch size of 1,024, and the rest with a batch size of 256.
• The sequence length was limited to 128 tokens for 90% of the steps and 512 for the remaining 10%.
• We use whole word masking and a duplicate factor of 10.
• Set max predictions per sequence to 20 for the dataset with max sequence length of 128 tokens and 80 for the dataset with max sequence length of 512 tokens.
• Use a random seed of 12345, masked language model probability of 0.15, and short sequence probability of 0.1.
• The optimizer is Adam with a learning rate of 1e - 4, \(\beta_{1}=0.9\) and \(\beta_{2}=0.999\), a weight decay of 0.01, learning rate warm - up for 10,000 steps and linear decay of the learning rate after.

Evaluation results

We evaluate the pre - trained language models on five NLP tasks using 12 datasets. We use Hugging Face's transformers v3.1.0 along with PyTorch v1.5.1 for fine - tuning. The fine - tuning is done by adding a fully connected linear layer to the last hidden state, and we use \(F_{1}\) score as a metric for all tasks. The fine - tuning code is available [here](https://github.com/CAMeL - Lab/CAMeLBERT).

Results (Average)

[1]: Variant-wise-average refers to average over a group of tasks in the same language variant.

🔧 Technical Details

• Pre - training implementation: We use [the original implementation](https://github.com/google - research/bert) released by Google.
• Hyperparameters: Follow the original English BERT model's hyperparameters, with specific settings for batch size, sequence length, masking, etc.

📄 License

This project is licensed under the Apache - 2.0 license.

Acknowledgements

This research was supported with Cloud TPUs from Google’s TensorFlow Research Cloud (TFRC).

Citation

@inproceedings{inoue-etal-2021-interplay,
    title = "The Interplay of Variant, Size, and Task Type in {A}rabic Pre-trained Language Models",
    author = "Inoue, Go  and
      Alhafni, Bashar  and
      Baimukan, Nurpeiis  and
      Bouamor, Houda  and
      Habash, Nizar",
    booktitle = "Proceedings of the Sixth Arabic Natural Language Processing Workshop",
    month = apr,
    year = "2021",
    address = "Kyiv, Ukraine (Online)",
    publisher = "Association for Computational Linguistics",
    abstract = "In this paper, we explore the effects of language variants, data sizes, and fine-tuning task types in Arabic pre-trained language models. To do so, we build three pre-trained language models across three variants of Arabic: Modern Standard Arabic (MSA), dialectal Arabic, and classical Arabic, in addition to a fourth language model which is pre-trained on a mix of the three. We also examine the importance of pre-training data size by building additional models that are pre-trained on a scaled-down set of the MSA variant. We compare our different models to each other, as well as to eight publicly available models by fine-tuning them on five NLP tasks spanning 12 datasets. Our results suggest that the variant proximity of pre-training data to fine-tuning data is more important than the pre-training data size. We exploit this insight in defining an optimized system selection model for the studied tasks.",
}