Chemical Bert Uncased

🚀 BERT for Chemical Industry

A BERT - based language model tailored for the chemical industry, offering enhanced performance in chemical - related text analysis.

🚀 Quick Start

This is a BERT - based language model that undergoes further pre - training from the checkpoint of SciBERT. We utilized a corpus of over 40,000+ technical documents from the Chemical Industrial domain and combined it with 13,000 Wikipedia Chemistry articles. The corpus includes Safety Data Sheets and Products Information Documents, containing over 250,000+ tokens from the Chemical domain. It was pre - trained using MLM on over 9.2 million paragraphs.

Masked language modeling (MLM)

In MLM, given a sentence, the model randomly masks 15% of the words in the input. Then it processes the entire masked sentence through the model and predicts the masked words. This is different from traditional recurrent neural networks (RNNs) which usually process words sequentially, or autoregressive models like GPT that internally mask future tokens. MLM enables the model to learn a bidirectional representation of the sentence.

💻 Usage Examples

Basic Usage

from transformers import pipeline
fill_mask = pipeline(
    "fill-mask",
    model="recobo/chemical-bert-uncased",
    tokenizer="recobo/chemical-bert-uncased"
)
fill_mask("we create [MASK]")