CovidBERT-NLI Open-Source Model - Generate General Sentence Embedding Representations Based on CORD19 Data

Covidbert Nli

Developed by gsarti

A BERT model trained on the CORD19 coronavirus research paper dataset, fine-tuned for natural language inference tasks to generate universal sentence embeddings

Text Embedding #COVID-19 Research Semantic Understanding #NLI Fine-tuned Embeddings #Biomedical Text Processing

Downloads 26

Release Time : 3/2/2022

Model Overview

This model is pre-trained on the CORD19 dataset and fine-tuned on SNLI and MultiNLI datasets, specifically designed for generating semantic representations of COVID-19-related texts, suitable for tasks such as scientific literature retrieval and semantic similarity calculation.

Model Features

Coronavirus Domain Adaptation

Pre-trained on the CORD19 coronavirus research paper dataset, providing better representation capabilities for COVID-19-related texts

Natural Language Inference Fine-tuning

Fine-tuned on SNLI and MultiNLI datasets, optimizing sentence-level semantic representation capabilities

Efficient Training Configuration

Optimized with a batch size of 64, 23,000 training steps, and 1,450 warm-up steps, completing training in just 6 hours on a P100 GPU

Model Capabilities

Text Semantic Representation

Sentence Similarity Calculation

Scientific Literature Retrieval

Natural Language Inference

Use Cases

Scientific Literature Processing

COVID-19 Literature Semantic Search

COVID-19 research paper retrieval system based on semantic similarity

Applied in the COVID Semantic Browser project

Scientific Literature Classification

Automatic classification of coronavirus-related research papers

🚀 CovidBERT-NLI

This is the CovidBERT model trained by DeepSet on AllenAI's CORD19 Dataset, which consists of scientific articles about coronaviruses. It aims to generate useful sentence embeddings for related research, leveraging pre - trained BERT and fine - tuning on specific datasets.

✨ Features

Utilizes the original BERT wordpiece vocabulary.
Fine - tuned on SNLI and MultiNLI datasets using the sentence - transformers library.
Employs the average pooling strategy and a softmax loss to produce universal sentence embeddings.

📦 Installation

No installation steps are provided in the original document.

💻 Usage Examples

An example usage for similarity - based scientific paper retrieval is provided in the Covid - 19 Semantic Browser repository.

📚 Documentation

Model Information

Base model: deepset/covid_bert_base from HuggingFace's AutoModel.
Training time: ~6 hours on the NVIDIA Tesla P100 GPU provided in Kaggle Notebooks.

Parameter Details

Parameter	Value
Batch size	64
Training steps	23000
Warmup steps	1450
Lowercasing	True
Max. Seq. Length	128

Performance Evaluation

The performance was evaluated on the test portion of the STS dataset using Spearman rank correlation and compared to the performances of similar models obtained with the same procedure to verify its performances.

Model	Score
`covidbert - nli` (this)	67.52
`gsarti/biobert - nli`	73.40
`gsarti/scibert - nli`	74.50
`bert - base - nli - mean - tokens`[2]	77.12

🔧 Technical Details

The model uses the original BERT wordpiece vocabulary and was fine - tuned on specific datasets. Parameter details for the original training on CORD - 19 are available on [DeepSet's MLFlow](https://public - mlflow.deepset.ai/#/experiments/2/runs/ba27d00c30044ef6a33b1d307b4a6cba).

📄 References

[1] A. Conneau et al., [Supervised Learning of Universal Sentence Representations from Natural Language Inference Data](https://www.aclweb.org/anthology/D17 - 1070/)

[2] N. Reimers et I. Gurevych, [Sentence - BERT: Sentence Embeddings using Siamese BERT - Networks](https://www.aclweb.org/anthology/D19 - 1410/)

Featured Recommended AI Models

Empowering the Future, Your AI Solution Knowledge Base

English 简体中文繁體中文にほんご