Linkbert Base

🚀 LinkBERT-base

LinkBERT-base is a pre - trained model leveraging English Wikipedia articles and hyperlink information. It offers enhanced performance in various NLP tasks by capturing document links.

🚀 Quick Start

LinkBERT-base is a model pre - trained on English Wikipedia articles with hyperlink information. It was introduced in the paper LinkBERT: Pretraining Language Models with Document Links (ACL 2022). The code and data are available in this repository.

✨ Features

• LinkBERT is a transformer encoder (BERT - like) model pre - trained on a large document corpus. It improves upon BERT by capturing document links such as hyperlinks and citation links, incorporating knowledge across multiple documents.
• It can serve as a drop - in replacement for BERT. It performs better in general language understanding tasks (e.g., text classification), and is especially effective for knowledge - intensive tasks (e.g., question answering) and cross - document tasks (e.g., reading comprehension, document retrieval).

📦 Installation

No specific installation steps are provided in the original document.

💻 Usage Examples

Basic Usage

To use the model to get the features of a given text in PyTorch:

from transformers import AutoTokenizer, AutoModel
tokenizer = AutoTokenizer.from_pretrained('michiyasunaga/LinkBERT-base')
model = AutoModel.from_pretrained('michiyasunaga/LinkBERT-base')
inputs = tokenizer("Hello, my dog is cute", return_tensors="pt")
outputs = model(**inputs)
last_hidden_states = outputs.last_hidden_state

Advanced Usage

For fine - tuning, you can use this repository or follow any other BERT fine - tuning codebases.

📚 Documentation

Intended uses & limitations

The model can be used by fine - tuning on a downstream task, such as question answering, sequence classification, and token classification. You can also use the raw model for feature extraction (i.e., obtaining embeddings for input text).

Evaluation results

When fine - tuned on downstream tasks, LinkBERT achieves the following results.

General benchmarks (MRQA and GLUE):

🔧 Technical Details

LinkBERT is a transformer encoder (BERT - like) model. It is an improvement of BERT that newly captures document links such as hyperlinks and citation links to include knowledge that spans across multiple documents. Specifically, it was pretrained by feeding linked documents into the same language model context, besides a single document.

📄 License

The model is licensed under the Apache - 2.0 license.

📚 Additional Information

Tags

• bert
• exbert
• linkbert
• feature - extraction
• fill - mask
• question - answering
• text - classification
• token - classification

Datasets

• wikipedia
• bookcorpus

📖 Citation

If you find LinkBERT useful in your project, please cite the following:

@InProceedings{yasunaga2022linkbert,
  author =  {Michihiro Yasunaga and Jure Leskovec and Percy Liang},
  title =   {LinkBERT: Pretraining Language Models with Document Links},
  year =    {2022},  
  booktitle = {Association for Computational Linguistics (ACL)},  
}