Tookabert Base

🚀 TookaBERT Model

TookaBERT models are a family of encoder models designed for Persian language processing. They come in two sizes, base and large, and are pre - trained on over 500GB of diverse Persian data, covering various topics like news, blogs, forums, and books. These models are pre - trained with the MLM (WWM) objective using two context lengths.

🚀 Quick Start

Model Details

TookaBERT models are a family of encoder models trained on Persian in two sizes: base and large. These models were pre - trained on over 500GB of Persian data, including a wide variety of topics such as News, Blogs, Forums, Books, etc. They were pre - trained with the MLM (WWM) objective using two context lengths.

For more information, you can read our paper on arXiv.

✨ Features

• Trained on a large corpus of Persian data (over 500GB).
• Available in two sizes: base and large.
• Pre - trained with the MLM (WWM) objective.

💻 Usage Examples

Basic Usage

You can use this model directly for Masked Language Modeling using the provided code below.

from transformers import AutoTokenizer, AutoModelForMaskedLM

tokenizer = AutoTokenizer.from_pretrained("PartAI/TookaBERT-Base")
model = AutoModelForMaskedLM.from_pretrained("PartAI/TookaBERT-Base")

# prepare input
text = "شهر برلین در کشور <mask> واقع شده است."
encoded_input = tokenizer(text, return_tensors='pt')

# forward pass
output = model(**encoded_input)

Advanced Usage

It is also possible to use inference pipelines such as below.

from transformers import pipeline

inference_pipeline = pipeline('fill-mask', model="PartAI/TookaBERT-Base")
inference_pipeline("شهر برلین در کشور <mask> واقع شده است.")

You can use this model to fine - tune it over your dataset and prepare it for your task.

• DeepSentiPers (Sentiment Analysis)
• ParsiNLU - Multiple - choice (Multiple - choice)

📚 Documentation

Evaluation

TookaBERT models are evaluated on a wide range of NLP downstream tasks, such as Sentiment Analysis (SA), Text Classification, Multiple - choice, Question Answering, and Named Entity Recognition (NER). Here are some key performance results:

*Note: because of the randomness in the fine - tuning process, results with less than 1% differences are considered together.

📄 License

This model is licensed under the Apache - 2.0 license.

🔧 Technical Details

The model is trained on Persian data using the MLM (WWM) objective with two context lengths. It comes in two sizes, base and large, to suit different application requirements.

📞 Contact us

If you have any questions regarding this model, you can reach us via the community of the model in Hugging Face.