Medbert Base

🚀 medBERT-base

This repository houses a BERT-based model, medBERT-base, fine-tuned on the gayanin/pubmed-gastro-maskfilling dataset for Masked Language Modeling (MLM). It aims to predict masked tokens in medical and gastroenterological texts, enhancing the model's comprehension and generation of medical information in natural language scenarios.

✨ Features

• Based on bert-base-uncased, tailored for medical text Masked Language Modeling.
• Utilizes BERT's WordPiece tokenizer.

📦 Installation

Since this model uses the Hugging Face transformers library, you can install it via:

pip install transformers

💻 Usage Examples

Basic Usage

You can load the pre-trained medBERT-base model using the Hugging Face transformers library:

from transformers import BertTokenizer, BertForMaskedLM
import torch

tokenizer = BertTokenizer.from_pretrained('suayptalha/medBERT-base')
model = BertForMaskedLM.from_pretrained('suayptalha/medBERT-base').to("cuda")

input_text = "Response to neoadjuvant chemotherapy best predicts survival [MASK] curative resection of gastric cancer."
inputs = tokenizer(input_text, return_tensors='pt').to("cuda")

outputs = model(**inputs)

masked_index = (inputs['input_ids'][0] == tokenizer.mask_token_id).nonzero(as_tuple=True)[0].item()

top_k = 5
logits = outputs.logits[0, masked_index]
top_k_ids = torch.topk(logits, k=top_k).indices.tolist()
top_k_tokens = tokenizer.convert_ids_to_tokens(top_k_ids)

print("Top 5 prediction:")
for i, token in enumerate(top_k_tokens):
    print(f"{i + 1}: {token}")

Top 5 prediction: 1: from 2: of 3: after 4: by 5: through

Advanced Usage

To fine-tune the medBERT-base model on your own medical dataset, follow these steps:

1. Prepare your dataset (e.g., medical texts or gastroenterology-related information) in text format.
2. Tokenize the dataset and apply masking.
3. Train the model using the provided training loop.

Here's the training code: medBERT-base.ipynb

🔧 Technical Details

Model Architecture

• Base Model: bert-base-uncased
• Task: Masked Language Modeling (MLM) for medical texts
• Tokenizer: BERT's WordPiece tokenizer

Training Details

Hyperparameters

• Batch Size: 16
• Learning Rate: 5e-5
• Number of Epochs: 1
• Max Sequence Length: 512 tokens

Dataset

• Dataset Name: gayanin/pubmed-gastro-maskfilling
• Task: Masked Language Modeling (MLM) on medical texts

📄 License

This project is licensed under the apache-2.0 license.

Acknowledgements

• The gayanin/pubmed-gastro-maskfilling dataset is available on the Hugging Face dataset hub and provides a rich collection of medical and gastroenterology-related information for training.
• This model uses the Hugging Face transformers library, which is a state-of-the-art library for NLP models

Support: