đ GENERator-eukaryote-3b-base Model
GENERator is a generative genomic foundation model with a 98k base - pair context length and 3B parameters. It's trained on a vast eukaryotic DNA dataset, enabling enhanced understanding and generation across various organisms.
đ Quick Start
In this repository, we present GENERator, a generative genomic foundation model featuring a context length of 98k base pairs and 3B parameters, trained on an expansive dataset comprising 386 billion base pairs of eukaryotic DNA. The extensive and diverse pre - training data endow the GENERator with enhanced understanding and generation capabilities across various organisms.
For more technical details, please refer to our paper GENERator: A Long - Context Generative Genomic Foundation Model. The code and implementation details are available on Github: https://github.com/GenerTeam/GENERator.
đģ Usage Examples
Basic Usage
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
tokenizer = AutoTokenizer.from_pretrained("GenerTeam/GENERator-eukaryote-3b-base", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("GenerTeam/GENERator-eukaryote-3b-base")
config = model.config
max_length = config.max_position_embeddings
sequences = [
"ATGAGGTGGCAAGAAATGGGCTAC",
"GAATTCCATGAGGCTATAGAATAATCTAAGAGAAAT"
]
sequences = [tokenizer.bos_token + sequence for sequence in sequences]
tokenizer.padding_side = "left"
inputs = tokenizer(
sequences,
add_special_tokens=False,
return_tensors="pt",
padding=True,
truncation=True,
max_length=max_length
)
with torch.inference_mode():
outputs = model.generate(**inputs, max_new_tokens=32, temperature=0.00001, top_k=1)
decoded_sequences = tokenizer.batch_decode(outputs, skip_special_tokens=True)
print(decoded_sequences)
Advanced Usage
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
tokenizer = AutoTokenizer.from_pretrained("GENERator-eukaryote-3b-base", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("GenerTeam/GENERator-eukaryote-3b-base")
config = model.config
max_length = config.max_position_embeddings
sequences = [
"ATGAGGTGGCAAGAAATGGGCTAC",
"GAATTCCATGAGGCTATAGAATAATCTAAGAGAAAT"
]
tokenizer.padding_side = "right"
inputs = tokenizer(
sequences,
add_special_tokens=True,
return_tensors="pt",
padding=True,
truncation=True,
max_length=max_length
)
with torch.inference_mode():
outputs = model(**inputs, output_hidden_states=True)
hidden_states = outputs.hidden_states[-1]
attention_mask = inputs["attention_mask"]
last_token_indices = attention_mask.sum(dim=1) - 1
seq_embeddings = []
for i, token_index in enumerate(last_token_indices):
seq_embedding = hidden_states[i, token_index, :]
seq_embeddings.append(seq_embedding)
seq_embeddings = torch.stack(seq_embeddings)
print("Sequence Embeddings:", seq_embeddings)
đ License
The project uses the MIT license.
đ Citation
@misc{wu2025generator,
title={GENERator: A Long-Context Generative Genomic Foundation Model},
author={Wei Wu and Qiuyi Li and Mingyang Li and Kun Fu and Fuli Feng and Jieping Ye and Hui Xiong and Zheng Wang},
year={2025},
eprint={2502.07272},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2502.07272},
}