ClinicalBERT Open-source Clinical Language Model - Free Support for High-efficiency Analysis of Electronic Health Records

Clinicalbert

Developed by medicalai

Clinical language model trained on 1.2 billion words of multicenter medical data, supporting electronic health record analysis

Large Language Model

Transformers

#Electronic Medical Record Analysis #Multicenter Medical Data #Disease Prediction

Downloads 163.90k

Release Time : 3/19/2023

Model Overview

ClinicalBERT is a BERT variant trained on large-scale medical texts, specifically designed for processing clinical electronic health records (EHR) and medical text analysis

Model Features

Large-scale medical data training

Trained on a 1.2 billion-word diverse disease corpus, covering a wide range of medical scenarios

Electronic health record optimization

Fine-tuned with over 3 million patient EHR data, specifically optimized for clinical document processing

Multicenter data sources

Training data sourced from multiple medical centers to ensure model generalization

Model Capabilities

Medical text understanding

Clinical entity recognition

Electronic health record analysis

Medical terminology standardization

Use Cases

Clinical decision support

Diabetes management assistance

Analyzing patient EHR data to assist in type 2 diabetes blood glucose control decisions

Showed effectiveness in optimizing blood glucose control in validation trials

Disease diagnosis assistance

General disease diagnosis

Providing diagnostic suggestions based on patient symptoms and medical history

Performed excellently in multiple disease diagnosis tasks

🚀 ClinicalBERT

ClinicalBERT is a model trained on a large multicenter dataset with a 1.2B - word corpus of diverse diseases. It's further fine - tuned on a large - scale EHR corpus from over 3 million patient records, aiming to provide better performance in medical - related tasks.

🚀 Quick Start

Load the model via the transformers library:

from transformers import AutoTokenizer, AutoModel
tokenizer = AutoTokenizer.from_pretrained("medicalai/ClinicalBERT")
model = AutoModel.from_pretrained("medicalai/ClinicalBERT")

✨ Features

This model card describes the ClinicalBERT model, which was trained on a large multicenter dataset with a large corpus of 1.2B words of diverse diseases we constructed. We then utilized a large - scale corpus of EHRs from over 3 million patient records to fine - tune the base language model.

📦 Installation

No specific installation steps other than the code for loading the model are provided in the original document, so this section is skipped.

💻 Usage Examples

Basic Usage

from transformers import AutoTokenizer, AutoModel
tokenizer = AutoTokenizer.from_pretrained("medicalai/ClinicalBERT")
model = AutoModel.from_pretrained("medicalai/ClinicalBERT")

📚 Documentation

Pretraining Data

The ClinicalBERT model was trained on a large multicenter dataset with a large corpus of 1.2B words of diverse diseases we constructed.

Model Pretraining

Pretraining Procedures

The ClinicalBERT was initialized from BERT. Then the training followed the principle of masked language model, in which given a piece of text, we randomly replace some tokens by MASKs, special tokens for masking, and then require the model to predict the original tokens via contextual text.

Pretraining Hyperparameters

We used a batch size of 32, a maximum sequence length of 256, and a learning rate of 5e - 5 for pre - training our models.

📄 License

No license information is provided in the original document, so this section is skipped.

📚 Citation

Please cite these articles:

Wang, G., Liu, X., Ying, Z. et al. Optimized glycemic control of type 2 diabetes with reinforcement learning: a proof - of - concept trial. Nat Med (2023). https://doi.org/10.1038/s41591 - 023 - 02552 - 9
Wang, G., Liu, X., Liu, H., Yang, G. et al. A Generalist Medical Language Model for Disease Diagnosis Assistance. Nat Med (2025). https://doi.org/10.1038/s41591 - 024 - 03416 - 6

Featured Recommended AI Models

Empowering the Future, Your AI Solution Knowledge Base

English 简体中文繁體中文にほんご