Chemberta Zinc Base V1

🚀 ChemBERTa: Training a BERT-like transformer model for masked language modelling of chemical SMILES strings

Deep learning in chemistry and materials science is a novel field full of potential. However, transfer - learning - based methods, popular in NLP and computer vision, haven't been effectively developed in computational chemistry and machine learning. Using HuggingFace's models and the ByteLevel tokenizer, we trained on a large corpus of 100k SMILES strings from the ZINC benchmark dataset.

🚀 Quick Start

The project's repository, which includes training, uploading, and evaluation notebooks (with sample predictions on compounds like Remdesivir), can be found here. All notebooks can be copied into a new Colab runtime for easy execution.

✨ Features

Model Training

• We trained a RoBERTa model over 5 epochs on 100k SMILES strings from the ZINC dataset. The model achieved a loss of 0.398, and the loss may continue to decline with more epochs.

Token Prediction

• The model can predict tokens within a SMILES sequence or molecule, enabling the prediction of molecule variants within the discoverable chemical space.

Problem - Solving

• By applying the learned representations of functional groups and atoms, we can address problems such as toxicity, solubility, drug - likeness, and synthesis accessibility on smaller datasets. These representations can be used as features for graph convolution and attention models on the graph structure of molecules, as well as for fine - tuning BERT.

Visualization

• Attention visualization can help chemistry practitioners and students quickly identify important sub - structures in various chemical properties. Previous research has shown that visualizing the attention mechanism is valuable for chemical reaction classification. Open - sourcing large - scale transformer models like RoBERTa with HuggingFace may accelerate related research.