# Masked language modeling

ChineseBERT is a Chinese pre-trained model that integrates glyph and pinyin information, optimized for Chinese text processing.

SyllaBERTa is an experimental Transformer-based masked language model specifically designed for processing Ancient Greek texts, employing syllable-level tokenization.

ModernCamemBERT is a French language model pre-trained on a 1T high-quality French text corpus. It is the French version of ModernBERT, focusing on long contexts and efficient inference speed.

RNA foundation model pre-trained on non-coding RNA data using masked language modeling (MLM) objective

medBERT-base is a BERT-based model focused on masked language modeling tasks for medical and gastroenterology texts.

A fine-tuned version based on the XLM-Roberta base model, using RoPE (Rotary Position Embedding) to replace the original positional embeddings, supporting 2048 sequence length

Ankh3 is a protein language model based on the T5 architecture. It is pre - trained by jointly optimizing masked language modeling and sequence completion tasks, and is suitable for protein feature extraction and sequence analysis.

RiNALMo is a non-coding RNA (ncRNA) model pre-trained based on the masked language modeling (MLM) objective, trained through self-supervised learning on a large number of non-coding RNA sequences.

Caduceus-PS is a DNA sequence modeling model with reverse-complement equivariance, designed for processing long sequences.

A multilingual ALBERT model pretrained with masked language modeling (MLM) objective, supporting 60+ languages, featuring a lightweight architecture with parameter sharing

Multilingual ALBERT model pretrained with masked language modeling objective, supporting 50+ languages, case-sensitive

Albertina 1.5B PTBR is a foundational large language model for the Brazilian Portuguese variant. It is an encoder belonging to the BERT family, based on the Transformer neural network architecture and developed on the basis of the DeBERTa model.

TahrirchiBERT-base is an encoder-only Transformer text model for Uzbek (Latin script) with 110 million parameters, pre-trained using masked language modeling objectives.

State-of-the-art BERT language model suite for Modern Hebrew

BERT model trained on German legal data, specialized in legal text processing

BEREL 3.0 is an embedding model based on the BERT architecture, specifically designed for rabbinic coded language, providing support for relevant research and applications.

BERTikal is a case-sensitive BERT base model for Brazilian legal language, trained on Brazilian legal texts and based on the BERTimbau checkpoint.

A RoBERTa-based masked language model specifically pretrained for news text

ARBERTv2 is an upgraded BERT model trained on Modern Standard Arabic (MSA) with a corpus of 243GB text, containing 27.8 billion tokens.

NorBERT 3 is a next-generation Norwegian language model based on the BERT architecture, supporting both Bokmål and Nynorsk written Norwegian.

A Mixture of Experts (MoE) model trained on masked language modeling tasks, with a parameter scale of 1.6 trillion. It uses an architecture similar to T5 but replaces the feed - forward layer with a sparse MLP layer.

This is a Cantonese model based on the base version of BART, obtained through second-phase pre-training on the LIHKG dataset.

ESM-2 is a next-generation protein model trained with masked language modeling objectives, suitable for fine-tuning on various downstream tasks with protein sequences as input.

ESM-2 is a state-of-the-art protein model trained on masked language modeling objectives, suitable for fine-tuning on various protein sequence input tasks.

ESM-2 is a cutting-edge protein model trained on masked language modeling objectives, suitable for various protein sequence analysis tasks

ESM-2 is a next-generation protein model trained with masked language modeling objectives, suitable for fine-tuning on various protein sequence tasks.

This is a MicroBERT model for the Coptic language, pre-trained through masked language modeling and supervised XPOS tagging.

This model studies the effectiveness of masking 15% content in masked language modeling, employing pre-layer normalization techniques not currently supported by HuggingFace.

A model pre-trained on Bulgarian language using Masked Language Modeling (MLM) objective, case-sensitive.

A BERT base model for the English language, pre-trained using the Masked Language Modeling (MLM) objective, case-insensitive.

A RoBERTa model for Turkish language, pre-trained with morphological-level tokenization and masked language modeling objectives, suitable for Turkish NLP tasks.

A RoBERTa model based on Turkish, pre-trained with masked language modeling (MLM) objective, case-insensitive.

A RoBERTa model pre-trained on Turkish with masked language modeling (MLM) objective, case-insensitive, medium-sized architecture.

A Chinese RoBERTa model pre-trained on CLUECorpusSmall, with a parameter scale of 8 layers and 512 hidden units, supporting masked language modeling tasks.

A language model pre-trained on a large corpus of battery research papers, inherited from SciBERT-cased, specializing in battery domain text comprehension

A Chinese RoBERTa model pre-trained on CLUECorpusSmall, with a parameter scale of 8 layers and 512 hidden units.

UzRoBerta is a pre-trained Uzbek (Cyrillic script) model for masked language modeling and next sentence prediction.

Masked language modeling pretrained model based on Macedonian language training with case-sensitive processing

SportsBERT is a BERT model specialized in the sports domain, trained on a corpus of sports news, suitable for sports-related natural language processing tasks.

An Indonesian pretrained model based on RoBERTa-base architecture, trained on Indonesian Wikipedia data, case insensitive.