Olmo2 8B SuperBPE T160k

🚀 SuperBPE

This 8B model is trained from scratch using a SuperBPE tokenizer, offering 30% more inference-time efficiency compared to BPE-trained models.

🚀 Quick Start

The 8B SuperBPE model is a remarkable innovation in the field of language models. It is trained from scratch with a SuperBPE tokenizer. SuperBPE is an extension of the BPE algorithm. It includes both traditional subword tokens (contained within word boundaries) and new superword tokens (containing parts of multiple words). Thanks to encoding the same amount of text in fewer tokens, this model is 30% more efficient at inference-time on average compared to a model trained with BPE.

The model is trained with the Olmo2 7B architecture and pretraining data. It has a context length of 2,884 tokens (to match the effective context size in bytes of a BPE model with a context length of 4,096 tokens) and is trained on 332B tokens. The tokenizer has a vocabulary size of 200k and transitions from learning subword to learning superword tokens at a vocabulary size of 160k.

💻 Usage Examples

Basic Usage

from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("UW/OLMo2-8B-SuperBPE-t160k")
model = AutoModelForCausalLM.from_pretrained("UW/OLMo2-8B-SuperBPE-t160k")

tokenizer.convert_ids_to_tokens(tokenizer.encode("By the way, I am a fan of the Milky Way."))
# ['ByĠtheĠway', ',ĠIĠamĠa', 'ĠfanĠofĠthe', 'ĠMilkyĠWay', '.']

📄 License

This project is licensed under the Apache-2.0 license.

📚 Documentation

Citation

@misc{liu-etal-2025-superbpe,
  title={SuperBPE: Space Travel for Language Models}, 
  author={Alisa Liu and Jonathan Hayase and Valentin Hofmann and Sewoong Oh and Noah A. Smith and Yejin Choi},
  year={2025},
  eprint={2503.13423},
  archivePrefix={arXiv},
  primaryClass={cs.CL},
  url={https://arxiv.org/abs/2503.13423}, 
}