Biolinkbert Large

🚀 BioLinkBERT-large

BioLinkBERT-large is a pre-trained model on PubMed abstracts with citation link information. It offers advanced performance in biomedical NLP tasks.

🚀 Quick Start

BioLinkBERT-large model is pretrained on PubMed abstracts along with citation link information. It is introduced in the paper LinkBERT: Pretraining Language Models with Document Links (ACL 2022). The code and data are available in this repository.

This model achieves state-of-the-art performance on several biomedical NLP benchmarks such as BLURB and MedQA-USMLE.

✨ Features

• Pretrained on a large corpus of documents with document links captured.
• Can be a drop-in replacement for BERT, achieving better performance in general language understanding tasks.
• Particularly effective for knowledge-intensive and cross-document tasks.

📦 Installation

The installation mainly involves using the transformers library. You can install it via pip install transformers if you haven't.

💻 Usage Examples

Basic Usage

To use the model to get the features of a given text in PyTorch:

from transformers import AutoTokenizer, AutoModel
tokenizer = AutoTokenizer.from_pretrained('michiyasunaga/BioLinkBERT-large')
model = AutoModel.from_pretrained('michiyasunaga/BioLinkBERT-large')
inputs = tokenizer("Sunitinib is a tyrosine kinase inhibitor", return_tensors="pt")
outputs = model(**inputs)
last_hidden_states = outputs.last_hidden_state

Advanced Usage

For fine-tuning, you can use this repository or follow any other BERT fine-tuning codebases.

📚 Documentation

Model description

LinkBERT is a transformer encoder (BERT-like) model pretrained on a large corpus of documents. It is an improvement of BERT that newly captures document links such as hyperlinks and citation links to include knowledge that spans across multiple documents. Specifically, it was pretrained by feeding linked documents into the same language model context, besides a single document.

LinkBERT can be used as a drop-in replacement for BERT. It achieves better performance for general language understanding tasks (e.g. text classification), and is also particularly effective for knowledge-intensive tasks (e.g. question answering) and cross-document tasks (e.g. reading comprehension, document retrieval).

Intended uses & limitations

The model can be used by fine-tuning on a downstream task, such as question answering, sequence classification, and token classification. You can also use the raw model for feature extraction (i.e. obtaining embeddings for input text).

🔧 Technical Details

LinkBERT is pretrained by incorporating document links into the training process, which is an innovative approach compared to traditional BERT. By feeding linked documents into the same language model context, it can capture knowledge across multiple documents, thus enhancing its performance in various tasks.

📄 License

This model is licensed under the Apache-2.0 license.

Evaluation results

When fine-tuned on downstream tasks, LinkBERT achieves the following results.

Biomedical benchmarks (BLURB, MedQA, MMLU, etc.): BioLinkBERT attains new state-of-the-art.

Citation

If you find LinkBERT useful in your project, please cite the following:

@InProceedings{yasunaga2022linkbert,
  author =  {Michihiro Yasunaga and Jure Leskovec and Percy Liang},
  title =   {LinkBERT: Pretraining Language Models with Document Links},
  year =    {2022},  
  booktitle = {Association for Computational Linguistics (ACL)},  
}

Additional Information

⚠️ Important Note

The model should be used in compliance with the Apache-2.0 license.

💡 Usage Tip

For better performance, fine-tuning on your specific downstream task is recommended.