Olmo 2 0425 1B

🚀 OLMo 2 1B

OLMo 2 1B is the smallest model in the OLMo 2 family, pre - trained on specific datasets to enable language model research.

🚀 Quick Start

You can quickly start using OLMo 2 1B by following the installation and inference steps below.

✨ Features

• Multiple Model Sizes: OLMo 2 family offers models in various sizes (1B, 7B, 13B, 32B) to meet different application needs.
• Diverse Training Data: Pre - trained on [OLMo - mix - 1124](https://huggingface.co/datasets/allenai/olmo - mix - 1124) and uses [Dolmino - mix - 1124](https://huggingface.co/datasets/allenai/dolmino - mix - 1124) for mid - training.
• Multiple Training Stages: Including initial pretraining, mid - training, and post - training with different techniques like SFT, DPO, and PPO.

📦 Installation

OLMo 2 1B is supported in transformers v4.48 or higher:

pip install transformers>=4.48

If using vLLM, you will need to install from the main branch until v0.7.4 is released.

💻 Usage Examples

Basic Usage

You can use OLMo with the standard HuggingFace transformers library:

from transformers import AutoModelForCausalLM, AutoTokenizer
olmo = AutoModelForCausalLM.from_pretrained("allenai/OLMo-2-0425-1B")
tokenizer = AutoTokenizer.from_pretrained("allenai/OLMo-2-0425-1B")
message = ["Language modeling is "]
inputs = tokenizer(message, return_tensors='pt', return_token_type_ids=False)
# optional verifying cuda
# inputs = {k: v.to('cuda') for k,v in inputs.items()}
# olmo = olmo.to('cuda')
response = olmo.generate(**inputs, max_new_tokens=100, do_sample=True, top_k=50, top_p=0.95)
print(tokenizer.batch_decode(response, skip_special_tokens=True)[0])
>> 'Language modeling is  a key component of any text-based application, but its effectiveness...'

Advanced Usage

For faster performance, you can quantize the model using the following method:

AutoModelForCausalLM.from_pretrained("allenai/OLMo-2-0425-1B",
    torch_dtype=torch.float16,
    load_in_8bit=True)  # Requires bitsandbytes

The quantized model is more sensitive to data types and CUDA operations. To avoid potential issues, it's recommended to pass the inputs directly to CUDA using:

inputs.input_ids.to('cuda')

To load a specific model revision with HuggingFace, simply add the argument revision:

olmo = AutoModelForCausalLM.from_pretrained("allenai/OLMo-2-0425-1B", revision="stage1-step140000-tokens294B")

Or, you can access all the revisions for the models via the following code snippet:

from huggingface_hub import list_repo_refs
out = list_repo_refs("allenai/OLMo-2-0425-1B")
branches = [b.name for b in out.branches]

Fine - tuning

Model fine - tuning can be done from the final checkpoint (the main revision of this model) or many intermediate checkpoints. Two recipes for tuning are available.

1. Fine - tune with the OLMo repository:

torchrun --nproc_per_node=8 scripts/train.py {path_to_train_config} \
    --data.paths=[{path_to_data}/input_ids.npy] \
    --data.label_mask_paths=[{path_to_data}/label_mask.npy] \
    --load_path={path_to_checkpoint} \
    --reset_trainer_state

For more documentation, see the GitHub README.

2. Further fine - tuning support is being developing in AI2's Open Instruct repository. Details are [here](https://github.com/allenai/open - instruct).

📚 Documentation

Model Description

Model Sources

• Project Page: https://allenai.org/olmo
• Repositories:
  • Core repo (training, inference, fine - tuning etc.): https://github.com/allenai/OLMo
  • Evaluation code: https://github.com/allenai/OLMo - Eval
  • Further fine - tuning code: https://github.com/allenai/open - instruct
• Paper: https://arxiv.org/abs/2501.00656

Evaluation

Core model results for OLMo 2 1B are found below.

Model Details

Training

Stage 1: Initial Pretraining

• Dataset: [OLMo - mix - 1124](https://huggingface.co/datasets/allenai/olmo - mix - 1124) (3.9T tokens)
• Coverage: 95%+ of total pretraining budget
• 1B Model: ~1 epoch

Stage 2: Mid - training

• Dataset: Dolmino - Mix - 1124
• One training mix:
  • 50B tokens
• Mix composition: 50% high - quality web data + academic/Q&A/instruction/math content

Model Merging

• 1B Model: only 1 version is trained on a 50B mix, we did not merge.

🔧 Technical Details

The OLMo 2 models are Transformer - style autoregressive language models. They are pre - trained on large - scale datasets and go through multiple training stages to improve performance. The models are designed to handle various natural language processing tasks.

📄 License

The code and model are released under Apache 2.0.

⚠️ Important Note

Like any base or fine - tuned language model, AI can be prompted by users to generate harmful and sensitive content. Such content may also be produced unintentionally, especially in cases involving bias, so we recommend that users consider the risks when applying this technology. Additionally, many statements from OLMo or any LLM are often inaccurate, so facts should be verified.

💡 Usage Tip

When using the quantized model, be aware of its sensitivity to data types and CUDA operations. Passing the inputs directly to CUDA can help avoid potential issues.

Citation

@misc{olmo20242olmo2furious,
      title={{2 OLMo 2 Furious}},
      author={Team OLMo and Pete Walsh and Luca Soldaini and Dirk Groeneveld and Kyle Lo and Shane Arora and Akshita Bhagia and Yuling Gu and Shengyi Huang and Matt Jordan and Nathan Lambert and Dustin Schwenk and Oyvind Tafjord and Taira Anderson and David Atkinson and Faeze Brahman and Christopher Clark and Pradeep Dasigi and Nouha Dziri and Michal Guerquin and Hamish Ivison and Pang Wei Koh and Jiacheng Liu and Saumya Malik and William Merrill and Lester James V. Miranda and Jacob Morrison and Tyler Murray and Crystal Nam and Valentina Pyatkin and Aman Rangapur and Michael Schmitz and Sam Skjonsberg and David Wadden and Christopher Wilhelm and Michael Wilson and Luke Zettlemoyer and Ali Farhadi and Noah A. Smith and Hannaneh Hajishirzi},
      year={2024},
      eprint={2501.00656},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2501.00656},
}

Model Card Contact

For errors in this model card, contact olmo@allenai.org.