Bioclinical ModernBERT Base

🚀 BioClinical ModernBERT

BioClinical ModernBERT offers two sizes: base (150M parameters) and large (396M parameters). You can find the model training checkpoints here, and our code is available in our GitHub repository.

🚀 Quick Start

BioClinical ModernBERT is a domain - adapted encoder based on ModernBERT. It can be used directly with the transformers library starting from v4.48.0. You can install the required library using the following command:

pip install -U transformers>=4.48.0

✨ Features

• Two Sizes: Available in base (150M parameters) and large (396M parameters) versions.
• Long - Context Processing: Incorporates long - context processing, which is beneficial for biomedical and clinical NLP tasks.
• Trained on Large Corpus: Trained on over 53.5 billion tokens from the largest biomedical and clinical corpus to date.
• Diverse Data Sources: Leverages 20 datasets from diverse institutions, domains, and geographic regions.

📦 Installation

You can install the necessary transformers library with the following command:

pip install -U transformers>=4.48.0

💻 Usage Examples

Basic Usage

Since BioClinical ModernBERT is a Masked Language Model (MLM), you can use the fill - mask pipeline or load it via AutoModelForMaskedLM.

Using AutoModelForMaskedLM:

from transformers import AutoTokenizer, AutoModelForMaskedLM
model_id = "thomas-sounack/BioClinical-ModernBERT-base"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForMaskedLM.from_pretrained(model_id)
text = "Mitochondria is the powerhouse of the [MASK]."
inputs = tokenizer(text, return_tensors="pt")
outputs = model(**inputs)
# To get predictions for the mask:
masked_index = inputs["input_ids"][0].tolist().index(tokenizer.mask_token_id)
predicted_token_id = outputs.logits[0, masked_index].argmax(axis=-1)
predicted_token = tokenizer.decode(predicted_token_id)
print("Predicted token:", predicted_token)
# Predicted token:  cell

Using a pipeline:

import torch
from transformers import pipeline
from pprint import pprint
pipe = pipeline(
    "fill-mask",
    model="thomas-sounack/BioClinical-ModernBERT-base",
    torch_dtype=torch.bfloat16,
)
input_text = "[MASK] is a disease caused by an uncontrolled division of abnormal cells in a part of the body."
results = pipe(input_text)
pprint(results)

Advanced Usage

To use BioClinical ModernBERT for downstream tasks like classification, retrieval, or QA, fine - tune it following standard BERT fine - tuning recipes.

⚠️ Important Note

If your GPU supports it, we recommend using BioClinical ModernBERT with Flash Attention 2 to reach the highest efficiency. To do so, install Flash Attention as follows, then use the model as normal:

pip install flash-attn

💡 Usage Tip

BioClinical ModernBERT, similarly to ModernBERT, does not use token type IDs unlike some earlier BERT models. Most downstream usage is identical to standard BERT models on the Hugging Face Hub, except you can omit the token_type_ids parameter.

📚 Documentation

Model Summary

BioClinical ModernBERT builds on ModernBERT base and large. It brings long - context processing and significant improvements in speed and performance for biomedical and clinical NLP. By using 20 diverse datasets, it addresses a key limitation of prior clinical encoders.

Training

Data

BioClinical ModernBERT is trained on 50.7B tokens of biomedical text from PubMed and PMC, and 2.8B tokens of clinical text from 20 datasets. The details are shown in the following table:

Methodology

BioClinical ModernBERT base is trained in two phases. It is initialized from the last stable - phase checkpoint of ModernBERT base and trained with a learning rate of 3e - 4 and a batch size of 72.

• Phase 1: Train on 160.5B tokens from PubMed, PMC, and the 20 clinical datasets. The learning rate remains constant, and the masking probability is set at 30%.
• Phase 2: Train only on the 20 clinical datasets. The masking probability is reduced to 15%. The model is trained for 3 epochs with a 1 - sqrt learning rate decay.

Evaluation

The following table shows the evaluation results of BioClinical ModernBERT compared with other models:

📄 License

We release the BioClinical ModernBERT base and large model weights and training checkpoints under the MIT license.

📚 Documentation

If you use BioClinical ModernBERT in your work, please cite our preprint:

@misc{sounack2025bioclinicalmodernbertstateoftheartlongcontext,
      title={BioClinical ModernBERT: A State-of-the-Art Long-Context Encoder for Biomedical and Clinical NLP}, 
      author={Thomas Sounack and Joshua Davis and Brigitte Durieux and Antoine Chaffin and Tom J. Pollard and Eric Lehman and Alistair E. W. Johnson and Matthew McDermott and Tristan Naumann and Charlotta Lindvall},
      year={2025},
      eprint={2506.10896},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2506.10896}, 
}