Open-source gpt2_zinc_87m model - Free generation of drug-like molecules and embedding representations of SMILES strings

Gpt2 Zinc 87m

Developed by entropy

An autoregressive language model based on GPT2 architecture, specifically designed for generating drug-like molecules or embedding representations from SMILES strings

Molecular Model

Transformers

Open Source License:MIT #Drug molecule generation #SMILES embedding representation #Autoregressive language model

Downloads 404

Release Time : 5/11/2023

Model Overview

This model was trained on approximately 480 million SMILES strings from the ZINC database, suitable for molecular generation tasks in chemistry and drug discovery

Model Features

Large-scale molecular training data

Trained on 480 million SMILES strings from the ZINC database

High generation quality

Generates molecules with high uniqueness and validity across different temperature settings

Embedding representation capability

Can generate meaningful embedding representations from SMILES strings

Optimized training

Trained for 175,000 iterations with a batch size of 3072, achieving a validation loss of approximately 0.615

Model Capabilities

Molecular generation

SMILES string embedding representation

Drug-like compound design

Use Cases

Drug discovery

Virtual compound library generation

Generate large numbers of potential drug candidate molecules

At temperature 1.0, generates 99.9% unique and 99.9% valid molecules

Molecular representation learning

Convert SMILES strings into embedding vectors for downstream tasks

Chemical research

Chemical space exploration

Generate novel molecular structures to explore chemical space

🚀 GPT2 Zinc 87m

This is a GPT2-style autoregressive language model trained on approximately 480 million SMILES strings from the ZINC database. It can generate druglike molecules or create embeddings from SMILES strings.

🚀 Quick Start

This GPT2-style autoregressive language model is trained on about 480m SMILES strings from the ZINC database. It has approximately 87m parameters and was trained for 175,000 iterations with a batch size of 3072, achieving a validation loss of around 0.615. This model is useful for generating druglike molecules or generating embeddings from SMILES strings.

💻 Usage Examples

Basic Usage

from transformers import GPT2TokenizerFast, GPT2LMHeadModel

tokenizer = GPT2TokenizerFast.from_pretrained("entropy/gpt2_zinc_87m", max_len=256)
model = GPT2LMHeadModel.from_pretrained('entropy/gpt2_zinc_87m')

Advanced Usage

Generate Molecules

inputs = torch.tensor([[tokenizer.bos_token_id]])

gen = model.generate(
              inputs,
              do_sample=True, 
              max_length=256, 
              temperature=1.,
              early_stopping=True,
              pad_token_id=tokenizer.pad_token_id,
              num_return_sequences=32
                         )
smiles = tokenizer.batch_decode(gen, skip_special_tokens=True)

Compute Embeddings

from transformers import DataCollatorWithPadding

collator = DataCollatorWithPadding(tokenizer, padding=True, return_tensors='pt')

inputs = collator(tokenizer(smiles))
outputs = model(**inputs, output_hidden_states=True)
full_embeddings = outputs[-1][-1]
mask = inputs['attention_mask']
embeddings = ((full_embeddings * mask.unsqueeze(-1)).sum(1) / mask.sum(-1).unsqueeze(-1))

⚠️ Important Note

This model was trained with bos and eos tokens around SMILES inputs. The GPT2TokenizerFast tokenizer DOES NOT ADD special tokens, even when add_special_tokens=True. Huggingface says this is intended behavior. It may be necessary to manually add these tokens.

inputs = collator(tokenizer([tokenizer.bos_token+i+tokenizer.eos_token for i in smiles]))

📚 Documentation

Model Performance

To test generation performance, 1 million compounds were generated at various temperature values. Generated compounds were checked for uniqueness and structural validity.

Property	Details
`percent_unique`	`n_unique_smiles/n_total_smiles`
`percent_valid`	`n_valid_smiles/n_unique_smiles`
`percent_unique_and_valid`	`n_valid_smiles/n_total_smiles`

Temperature	Percent Unique	Percent Valid	Percent Unique and Valid
0.5	0.928074	1	0.928074
0.75	0.998468	0.999967	0.998436
1	0.999659	0.999164	0.998823
1.25	0.999514	0.99351	0.993027
1.5	0.998749	0.970223	0.96901

Property histograms computed over 1 million generated compounds:

📄 License

This project is licensed under the MIT license.

Featured Recommended AI Models

Empowering the Future, Your AI Solution Knowledge Base

English 简体中文繁體中文にほんご