Medbert

🚀 MedBERT Model

MedBERT is a newly pre - trained transformer - based language model for biomedical named entity recognition. It is initialized with Bio_ClinicalBERT and pre - trained on N2C2, BioNLP, and CRAFT community datasets, aiming to solve the problem of named entity recognition in the biomedical field and provide more accurate information extraction for the biomedical industry.

🚀 Quick Start

from transformers import AutoTokenizer, AutoModel
tokenizer = AutoTokenizer.from_pretrained("Charangan/MedBERT")
model = AutoModel.from_pretrained("Charangan/MedBERT")

✨ Features

• Initialized with Bio_ClinicalBERT, leveraging pre - trained knowledge.
• Pre - trained on multiple community datasets, enhancing generalization ability in the biomedical domain.

📦 Installation

The installation mainly involves using the transformers library. You can install it via the following command:

pip install transformers

💻 Usage Examples

Basic Usage

from transformers import AutoTokenizer, AutoModel
tokenizer = AutoTokenizer.from_pretrained("Charangan/MedBERT")
model = AutoModel.from_pretrained("Charangan/MedBERT")

📚 Documentation

Pretraining

Data

The MedBERT model was trained on N2C2, BioNLP, and CRAFT community datasets, along with Wikipedia - crawled medical - related articles.

Procedures

The model was trained using code from [Google's BERT repository](https://github.com/google - research/bert). Model parameters were initialized with Bio_ClinicalBERT.

Hyperparameters

We used a batch size of 32, a maximum sequence length of 256, and a learning rate of 1·10−4 for pre - training our models. The models trained for 200,000 steps. The dup factor for duplicating input data with different masks was set to 5. All other default parameters were used (specifically, masked language model probability = 0.15 and max predictions per sequence = 22).

More Information

Refer to the original paper, MedBERT: A Pre - trained Language Model for Biomedical Named Entity Recognition (APSIPA Conference 2022) for additional details and performance of biomedical NER tasks.

📄 License

This project is licensed under the MIT License.

📚 Citation

@INPROCEEDINGS{9980157,
    author={Vasantharajan, Charangan and Tun, Kyaw Zin and Thi - Nga, Ho and Jain, Sparsh and Rong, Tong and Siong, Chng Eng},
    booktitle={2022 Asia - Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC)},
    title={MedBERT: A Pre - trained Language Model for Biomedical Named Entity Recognition},
    year={2022},
    volume={},
    number={},
    pages={1482 - 1488},
    doi={10.23919/APSIPAASC55919.2022.9980157}
}