Gemma 2 9b It SimPO

🚀 gemma-2-9b-it-SimPO Model Card

SimPO (Simple Preference Optimization) is an offline preference optimization algorithm that enhances the training of large language models (LLMs) using preference optimization datasets. It aligns the reward function with the generation likelihood, eliminating the need for a reference model and incorporating a target reward margin to boost performance. For more details, refer to our preprint and github repo.

🚀 Quick Start

Here is an example code to get started with the model:

import torch
from transformers import pipeline

model_id = "princeton-nlp/gemma-2-9b-it-SimPO"

generator = pipeline(
    "text-generation",
    model=model_id,
    model_kwargs={"torch_dtype": torch.bfloat16},
    device="cuda",
)
outputs = generator([{"role": "user", "content": "What's the difference between llamas and alpacas?"}],
                      do_sample=False,
                      eos_token_id=[generator.tokenizer.convert_tokens_to_ids("<end_of_turn>"), generator.tokenizer.eos_token_id],
                      max_new_tokens=200)
print(outputs[0]['generated_text'])

✨ Features

SimPO is designed to optimize the training of large language models with preference optimization datasets, eliminating the need for a reference model and improving performance by aligning the reward function with the generation likelihood.

📦 Installation

No specific installation steps are provided in the original document, so this section is skipped.

💻 Usage Examples

Basic Usage

import torch
from transformers import pipeline

model_id = "princeton-nlp/gemma-2-9b-it-SimPO"

generator = pipeline(
    "text-generation",
    model=model_id,
    model_kwargs={"torch_dtype": torch.bfloat16},
    device="cuda",
)
outputs = generator([{"role": "user", "content": "What's the difference between llamas and alpacas?"}],
                      do_sample=False,
                      eos_token_id=[generator.tokenizer.convert_tokens_to_ids("<end_of_turn>"), generator.tokenizer.eos_token_id],
                      max_new_tokens=200)
print(outputs[0]['generated_text'])

📚 Documentation

Model Details

Model Description

We fine-tuned google/gemma-2-9b-it on princeton-nlp/gemma2-ultrafeedback-armorm with the SimPO objective.

Model Sources

• Repository: https://github.com/princeton-nlp/SimPO
• Paper: https://arxiv.org/pdf/2405.14734

Training Details

Training Data

We use princeton-nlp/gemma2-ultrafeedback-armorm as the preference optimization dataset.

Training Hyperparameters

The hyperparameters used can be found in the training script.

Speeds, Sizes, Times

Fine-tuning the google/gemma-2-9b-it on princeton-nlp/gemma2-ultrafeedback-armorm takes around 100 mins to finish on 8xH100 GPUs.

Evaluation Results

🔧 Technical Details

Model Architecture and Objective

The model architecture is based on google/gemma-2-9b-it. We use the SimPO training objective proposed in our preprint.

Hardware

We used 8xH100 GPUs for model training.

Software

Training was done using the alignment-handbook library.

📄 License

The model is released under the MIT license.

📖 Citation

gemma model

@article{gemma_2024,
    title={Gemma},
    url={https://www.kaggle.com/m/3301},
    DOI={10.34740/KAGGLE/M/3301},
    publisher={Kaggle},
    author={Gemma Team},
    year={2024}
}

SimPO paper

@article{meng2024simpo,
  title={{SimPO}: Simple preference optimization with a reference-free reward},
  author={Meng, Yu and Xia, Mengzhou and Chen, Danqi},
  journal={arXiv preprint arXiv:2405.14734},
  year={2024}
}

UltraFeedback paper

@article{cui2023ultrafeedback,
  title={{UltraFeedback}: Boosting language models with high-quality feedback},
  author={Cui, Ganqu and Yuan, Lifan and Ding, Ning and Yao, Guanming and Zhu, Wei and Ni, Yuan and Xie, Guotong and Liu, Zhiyuan and Sun, Maosong},
  journal={arXiv preprint arXiv:2310.01377},
  year={2023}
}

ArmoRM paper

@article{wang2024interpretable,
  title={Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts},
  author={Wang, Haoxiang and Xiong, Wei and Xie, Tengyang and Zhao, Han and Zhang, Tong},
  journal={arXiv preprint arXiv:2406.12845},
  year={2024}
}