Araelectra Base Generator

🚀 AraELECTRA

ELECTRA is a method for self - supervised language representation learning. It can pre - train transformer networks with relatively little compute. AraELECTRA models are trained to distinguish "real" input tokens from "fake" ones generated by another neural network, similar to the discriminator of a GAN. AraELECTRA achieves state - of - the - art results on Arabic QA datasets.

For a detailed description, please refer to the AraELECTRA paper AraELECTRA: Pre - Training Text Discriminators for Arabic Language Understanding.

🚀 Quick Start

💻 Usage Examples

Basic Usage

from transformers import pipeline

fill_mask = pipeline(
    "fill-mask",
    model="aubmindlab/araelectra-base-generator",
    tokenizer="aubmindlab/araelectra-base-generator"
)

print(
    fill_mask(" عاصمة لبنان هي [MASK] .")
)

📦 Installation

It is recommended to apply our preprocessing function before training/testing on any dataset.

Install the arabert python package to segment text for AraBERT v1 & v2 or to clean your data pip install arabert

from arabert.preprocess import ArabertPreprocessor

model_name="aubmindlab/araelectra-base"
arabert_prep = ArabertPreprocessor(model_name=model_name)

text = "ولن نبالغ إذا قلنا إن هاتف أو كمبيوتر المكتب في زمننا هذا ضروري"
arabert_prep.preprocess(text)

>>> output: ولن نبالغ إذا قلنا : إن هاتف أو كمبيوتر المكتب في زمننا هذا ضروري

📚 Documentation

Model

Compute

Dataset

The pretraining data used for the new AraELECTRA model is also used for AraGPT2 and AraELECTRA.

The dataset consists of 77GB or 200,095,961 lines or 8,655,948,860 words or 82,232,988,358 chars (before applying Farasa Segmentation)

For the new dataset, we added the unshuffled OSCAR corpus, after thorough filtering, to the previous dataset used in AraBERTv1 but without the websites that we previously crawled:

• OSCAR unshuffled and filtered.
• Arabic Wikipedia dump from 2020/09/01
• The 1.5B words Arabic Corpus
• The OSIAN Corpus
• Assafir news articles. Huge thank you to Assafir for providing the data

TensorFlow 1.x models

You can find the PyTorch, TF2 and TF1 models in HuggingFace's Transformer Library under the aubmindlab username

• wget https://huggingface.co/aubmindlab/MODEL_NAME/resolve/main/tf1_model.tar.gz where MODEL_NAME is any model under the aubmindlab name

📄 License

If you used this model, please cite us as:

@inproceedings{antoun-etal-2021-araelectra,
    title = "{A}ra{ELECTRA}: Pre-Training Text Discriminators for {A}rabic Language Understanding",
    author = "Antoun, Wissam  and
      Baly, Fady  and
      Hajj, Hazem",
    booktitle = "Proceedings of the Sixth Arabic Natural Language Processing Workshop",
    month = apr,
    year = "2021",
    address = "Kyiv, Ukraine (Virtual)",
    publisher = "Association for Computational Linguistics",
    url = "https://www.aclweb.org/anthology/2021.wanlp-1.20",
    pages = "191--195",
}

Acknowledgments

Thanks to TensorFlow Research Cloud (TFRC) for the free access to Cloud TPUs. We couldn't have done it without this program. Thanks also to the AUB MIND Lab Members for their continuous support. Additionally, thanks to Yakshof and Assafir for data and storage access. Another thanks to Habib Rahal (https://www.behance.net/rahalhabib) for putting a face to AraBERT.

Contacts

Wissam Antoun: Linkedin | Twitter | Github | wfa07@mail.aub.edu | wissam.antoun@gmail.com

Fady Baly: Linkedin | Twitter | Github | fgb06@mail.aub.edu | baly.fady@gmail.com