COVID SciBERT

🚀 COVID-SciBERT

COVID-SciBERT is a small language modelling expansion of SciBERT, a BERT model trained on scientific text.

🚀 Quick Start

To use the COVID-SciBERT model, you can follow the pipelining example below. It demonstrates how to load the model, tokenizer, and perform a fill-mask task.

import transformers

model = transformers.AutoModelWithLMHead.from_pretrained('lordtt13/COVID-SciBERT')

tokenizer = transformers.AutoTokenizer.from_pretrained('lordtt13/COVID-SciBERT')

nlp_fill = transformers.pipeline('fill-mask', model = model, tokenizer = tokenizer)
nlp_fill('Coronavirus or COVID-19 can be prevented by a' + nlp_fill.tokenizer.mask_token)

# Output:
# [{'sequence': '[CLS] coronavirus or covid - 19 can be prevented by a combination [SEP]',
#   'score': 0.1719885915517807,
#   'token': 2702},
#  {'sequence': '[CLS] coronavirus or covid - 19 can be prevented by a simple [SEP]',
#   'score': 0.054218728095293045,
#   'token': 2177},
#  {'sequence': '[CLS] coronavirus or covid - 19 can be prevented by a novel [SEP]',
#   'score': 0.043364267796278,
#   'token': 3045},
#  {'sequence': '[CLS] coronavirus or covid - 19 can be prevented by a high [SEP]',
#   'score': 0.03732519596815109,
#   'token': 597},
#  {'sequence': '[CLS] coronavirus or covid - 19 can be prevented by a vaccine [SEP]',
#   'score': 0.021863549947738647,
#   'token': 7039}]

✨ Features

• Based on SciBERT: COVID-SciBERT is an expansion of the SciBERT model, which is trained on scientific text.
• Expanded with COVID-19 data: The model is further trained using the abstracts from the CORD-19 dataset, making it more relevant to COVID-19 research.

📚 Documentation

Details of SciBERT

The SciBERT model was presented in SciBERT: A Pretrained Language Model for Scientific Text by Iz Beltagy, Kyle Lo, Arman Cohan. Here is the abstract:

Obtaining large-scale annotated data for NLP tasks in the scientific domain is challenging and expensive. We release SciBERT, a pretrained language model based on BERT (Devlin et al., 2018) to address the lack of high-quality, large-scale labeled scientific data. SciBERT leverages unsupervised pretraining on a large multi-domain corpus of scientific publications to improve performance on downstream scientific NLP tasks. We evaluate on a suite of tasks including sequence tagging, sentence classification and dependency parsing, with datasets from a variety of scientific domains. We demonstrate statistically significant improvements over BERT and achieve new state-of-the-art results on several of these tasks.

Details of the downstream task (Language Modeling) - Dataset 📚

There are actually two datasets that have been used here:

• The original SciBERT model is trained on papers from the corpus of semanticscholar.org. Corpus size is 1.14M papers, 3.1B tokens. They used the full text of the papers in training, not just abstracts. SciBERT has its own vocabulary (scivocab) that's built to best match the training corpus.

• The expansion is done using the papers present in the COVID-19 Open Research Dataset Challenge (CORD-19). Only the abstracts have been used and vocabulary was pruned and added to the existing scivocab. In response to the COVID-19 pandemic, the White House and a coalition of leading research groups have prepared the COVID-19 Open Research Dataset (CORD-19). CORD-19 is a resource of over 200,000 scholarly articles, including over 100,000 with full text, about COVID-19, SARS-CoV-2, and related coronaviruses. This freely available dataset is provided to the global research community to apply recent advances in natural language processing and other AI techniques to generate new insights in support of the ongoing fight against this infectious disease. There is a growing urgency for these approaches because of the rapid acceleration in new coronavirus literature, making it difficult for the medical research community to keep up.

Model training

The training script is present here.

Created by Tanmay Thakur | LinkedIn

PS: Still looking for more resources to expand my expansion!