language:

• fa library_name: transformers widget:
  • text: "ز سوزناکی گفتار من [MASK] بگریست" example_title: "Poetry 1"
  • text: "نظر از تو برنگیرم همه [MASK] تا بمیرم که تو در دلم نشستی و سر مقام داری" example_title: "Poetry 2"
  • text: "هر ساعتم اندرون بجوشد [MASK] را وآگاهی نیست مردم بیرون را" example_title: "Poetry 3"
  • text: "غلام همت آن رند عافیت سوزم که در گدا صفتی [MASK] داند" example_title: "Poetry 4"
  • text: "این [MASK] اولشه." example_title: "Informal 1"
  • text: "دیگه خسته شدم! [MASK] اینم شد کار؟!" example_title: "Informal 2"
  • text: "فکر نکنم به موقع برسیم. بهتره [MASK] این یکی بشیم." example_title: "Informal 3"
  • text: "تا صبح بیدار موندم و داشتم برای [MASK] آماده می شدم." example_title: "Informal 4"
  • text: "زندگی بدون [MASK] خسته‌کننده است." example_title: "Formal 1"
  • text: "در حکم اولیه این شرکت مجاز به فعالیت شد ولی پس از بررسی مجدد، مجوز این شرکت [MASK] شد." example_title: "Formal 2"

FaBERT: Pre-training BERT on Persian Blogs

Model Details

FaBERT is a Persian BERT-base model trained on the diverse HmBlogs corpus, encompassing both casual and formal Persian texts. Developed for natural language processing tasks, FaBERT is a robust solution for processing Persian text. Through evaluation across various Natural Language Understanding (NLU) tasks, FaBERT consistently demonstrates notable improvements, while having a compact model size. Now available on Hugging Face, integrating FaBERT into your projects is hassle-free. Experience enhanced performance without added complexity as FaBERT tackles a variety of NLP tasks.

Features

• Pre-trained on the diverse HmBlogs corpus consisting more than 50 GB of text from Persian Blogs
• Remarkable performance across various downstream NLP tasks
• BERT architecture with 124 million parameters

Useful Links

• Repository: FaBERT on Github
• Paper: arXiv preprint

Usage

Loading the Model with MLM head

from transformers import AutoTokenizer, AutoModelForMaskedLM

tokenizer = AutoTokenizer.from_pretrained("sbunlp/fabert") # make sure to use the default fast tokenizer
model = AutoModelForMaskedLM.from_pretrained("sbunlp/fabert")

Downstream Tasks

Similar to the original English BERT, FaBERT can be fine-tuned on many downstream tasks.(https://huggingface.co/docs/transformers/en/training)

Examples on Persian datasets are available in our GitHub repository.

make sure to use the default Fast Tokenizer

Training Details

FaBERT was pre-trained with the MLM (WWM) objective, and the resulting perplexity on validation set was 7.76.

Evaluation

Here are some key performance results for the FaBERT model:

Sentiment Analysis

Named Entity Recognition

Question Answering

Natural Language Inference & QQP

Number of Parameters

For a more detailed performance analysis refer to the paper.

How to Cite

If you use FaBERT in your research or projects, please cite it using the following BibTeX:

@article{masumi2024fabert,
  title={FaBERT: Pre-training BERT on Persian Blogs},
  author={Masumi, Mostafa and Majd, Seyed Soroush and Shamsfard, Mehrnoush and Beigy, Hamid},
  journal={arXiv preprint arXiv:2402.06617},
  year={2024}
}