Scincl

🚀 SciNCL

SciNCL is a pre - trained BERT language model designed to generate document - level embeddings of research papers. It leverages the citation graph neighborhood for contrastive learning, offering valuable insights for scientific document analysis.

🚀 Quick Start

SciNCL is a pre - trained BERT language model to generate document - level embeddings of research papers. It uses the citation graph neighborhood to generate samples for contrastive learning. Prior to the contrastive training, the model is initialized with weights from scibert - scivocab - uncased. The underlying citation embeddings are trained on the S2ORC citation graph.

Paper: Neighborhood Contrastive Learning for Scientific Document Representations with Citation Embeddings (EMNLP 2022 paper).

Code: https://github.com/malteos/scincl

PubMedNCL: Working with biomedical papers? Try PubMedNCL.

✨ Features

• Document - level Embeddings: Capable of generating high - quality document - level embeddings for research papers.
• Contrastive Learning: Utilizes citation graph neighborhood for contrastive learning, enhancing the model's performance.
• Pre - trained Weights: Initialized with weights from scibert - scivocab - uncased for better generalization.

📦 Installation

The installation process is mainly about loading the model through the sentence - transformers or transformers library. There is no specific installation command here, but you need to ensure that the relevant libraries are installed. You can use the following command to install the necessary libraries:

pip install sentence-transformers transformers

💻 Usage Examples

Basic Usage

Sentence Transformers

from sentence_transformers import SentenceTransformer

# Load the model
model = SentenceTransformer("malteos/scincl")

# Concatenate the title and abstract with the [SEP] token
papers = [
    "BERT [SEP] We introduce a new language representation model called BERT",
    "Attention is all you need [SEP] The dominant sequence transduction models are based on complex recurrent or convolutional neural networks",
]
# Inference
embeddings = model.encode(papers)

# Compute the (cosine) similarity between embeddings
similarity = model.similarity(embeddings[0], embeddings[1])
print(similarity.item())
# => 0.8440517783164978

Transformers

from transformers import AutoTokenizer, AutoModel

# load model and tokenizer
tokenizer = AutoTokenizer.from_pretrained('malteos/scincl')
model = AutoModel.from_pretrained('malteos/scincl')

papers = [{'title': 'BERT', 'abstract': 'We introduce a new language representation model called BERT'},
          {'title': 'Attention is all you need', 'abstract': ' The dominant sequence transduction models are based on complex recurrent or convolutional neural networks'}]

# concatenate title and abstract with [SEP] token
title_abs = [d['title'] + tokenizer.sep_token + (d.get('abstract') or '') for d in papers]

# preprocess the input
inputs = tokenizer(title_abs, padding=True, truncation=True, return_tensors="pt", max_length=512)

# inference
result = model(**inputs)

# take the first token ([CLS] token) in the batch as the embedding
embeddings = result.last_hidden_state[:, 0, :]

# calculate the similarity
embeddings = torch.nn.functional.normalize(embeddings, p=2, dim=1)
similarity = (embeddings[0] @ embeddings[1].T)
print(similarity.item())
# => 0.8440518379211426

📚 Documentation

Triplet Mining Parameters

SciDocs Results

These model weights are the ones that yielded the best results on SciDocs (seed = 4). In the paper, the SciDocs results are reported as the mean over ten seeds.

📄 License

This project is licensed under the MIT license.