Tookabert Large

🚀 TookaBERT Model

TookaBERT models are encoder models designed for Persian language processing. They come in two sizes, base and large, and are pre - trained on over 500GB of Persian data covering diverse topics like news, blogs, forums, and books. These models are pre - trained with the MLM (WWM) objective using two context lengths. Notably, TookaBERT - Large is the first large encoder model pre - trained on Persian and currently stands as the state - of - the - art model for Persian tasks.

Model Metadata

Widget Examples

• "توانا بود هر که بود ز دانش دل پیر برنا بود"
• "شهر برلین در کشور واقع شده است."
• "بهنام از خوانندگان مشهور کشور ما است."
• "رضا از بازیگران مشهور کشور ما است."
• "سید ابراهیم رییسی در سال رییس جمهور ایران شد."
• "دیگر امکان ادامه وجود ندارد. باید قرارداد را کنیم."

🚀 Quick Start

✨ Features

• Two Sizes: Available in base and large sizes to suit different needs.
• Extensive Training Data: Pre - trained on over 500GB of diverse Persian data.
• State - of - the - Art: TookaBERT - Large is the leading model for Persian tasks.

📦 Installation

No specific installation steps are provided in the original document.

💻 Usage Examples

Basic Usage

You can use this model directly for Masked Language Modeling using the provided code below.

from transformers import AutoTokenizer, AutoModelForMaskedLM

tokenizer = AutoTokenizer.from_pretrained("PartAI/TookaBERT-Large")
model = AutoModelForMaskedLM.from_pretrained("PartAI/TookaBERT-Large")

# prepare input
text = "شهر برلین در کشور <mask> واقع شده است."
encoded_input = tokenizer(text, return_tensors='pt')

# forward pass
output = model(**encoded_input)

Advanced Usage

It is also possible to use inference pipelines such as below.

from transformers import pipeline

inference_pipeline = pipeline('fill-mask', model="PartAI/TookaBERT-Large")
inference_pipeline("شهر برلین در کشور <mask> واقع شده است.")

You can use this model to fine - tune it over your dataset and prepare it for your task.

• DeepSentiPers (Sentiment Analysis)
• ParsiNLU - Multiple - choice (Multiple - choice)

📚 Documentation

For more information you can read our paper on arXiv.

🔧 Technical Details

TookaBERT models are pre - trained on Persian data with the MLM (WWM) objective using two context lengths. The large version, TookaBERT - Large, is a significant milestone as the first large encoder model pre - trained on Persian.

📄 License

The model is licensed under the Apache - 2.0 license.

📊 Evaluation

TookaBERT models are evaluated on a wide range of NLP downstream tasks, such as Sentiment Analysis (SA), Text Classification, Multiple - choice, Question Answering, and Named Entity Recognition (NER). Here are some key performance results:

*Note: because of the randomness in the fine - tuning process, results with less than 1% differences are considered together.

📞 Contact us

If you have any questions regarding this model, you can reach us via the community of the model in Hugging Face.