Dziribert

🚀 DziriBERT

DziriBERT is the first Transformer-based Language Model pre - trained specifically for the Algerian Dialect. It can handle Algerian text written in both Arabic and Latin characters. Despite being pre - trained on relatively less data (about 1 million tweets), it achieves state - of - the - art results on Algerian text classification datasets.

For more details, check our paper: https://arxiv.org/pdf/2109.12346.pdf.

🚀 Quick Start

✨ Features

• Specifically pre - trained for the Algerian Dialect.
• Handles text in both Arabic and Latin characters.
• Sets new state - of - the - art results on Algerian text classification datasets.

📦 Installation

No specific installation steps provided in the original README.

💻 Usage Examples

Basic Usage

from transformers import BertTokenizer, BertForMaskedLM

tokenizer = BertTokenizer.from_pretrained("alger-ia/dziribert")
model = BertForMaskedLM.from_pretrained("alger-ia/dziribert")

You can find a fine - tuning script in our Github repo: https://github.com/alger-ia/dziribert

📚 Documentation

Limitations

The pre - training data used in this project comes from social media (Twitter). So, the Masked Language Modeling objective may predict offensive words in some cases. Modeling such words can be either beneficial (e.g., when training a hate speech model) or detrimental (e.g., when generating responses directly sent to the end - user). Depending on your downstream task, you may need to filter out these words, especially when providing automatically generated text to the end - user.

How to cite

@article{dziribert,
  title={DziriBERT: a Pre-trained Language Model for the Algerian Dialect},
  author={Abdaoui, Amine and Berrimi, Mohamed and Oussalah, Mourad and Moussaoui, Abdelouahab},
  journal={arXiv preprint arXiv:2109.12346},
  year={2021}
}

📄 License

This project is licensed under the Apache - 2.0 license.

Contact

Please contact amine.abdaoui.nlp@gmail.com for any questions, feedback, or requests.

Additional Information