Llama 3 8B SFR Iterative DPO R

🚀 Llama-3-8B-SFR-Iterative-DPO-R

We release a state-of-the-art instruct model that outperforms many models on multiple benchmarks and is trained with open-sourced datasets.

🚀 Quick Start

We introduce Llama-3-8B-SFR-Iterative-DPO-R, a cutting - edge instruct model. It excels on three popular instruct model benchmarks: Alpaca-Eval-V2, MT-Bench, and Chat-Arena-Hard, outperforming similar - sized models, most large open - sourced models, and strong proprietary models. This model is trained using open - sourced datasets without any additional human - or GPT4 - labeling.

✨ Features

• High - performance: Surpasses many models on various benchmarks.
• Cost - effective training: Utilizes a DPO - based online RLHF recipe, which is cheaper and simpler to train and tune compared to PPO - based approaches.
• Distribution shift mitigation: The online component effectively reduces distribution shifts during policy optimization.

📦 Installation

No installation steps are provided in the original document, so this section is skipped.

💻 Usage Examples

Basic Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

device = "cuda" 

model = AutoModelForCausalLM.from_pretrained("Salesforce/Llama-3-8B-SFR-Iterative-DPO-R")
tokenizer = AutoTokenizer.from_pretrained("Salesforce/Llama-3-8B-SFR-Iterative-DPO-R")

messages = [
    {"role": "user", "content": "I'm trying to teach myself to have nicer handwriting. Can you help?"},
]

model_inputs = tokenizer.apply_chat_template(messages, return_tensors="pt")

model_inputs = model_inputs.to(device)
model.to(device)

output_tokens = model.generate(model_inputs, max_new_tokens=1024, do_sample=True)
model_outputs = tokenizer.batch_decode(output_tokens)
print(model_outputs[0])

📚 Documentation

Model Releases

• SFT model
• Reward model
• RLHF model

Training methods

We have developed a simple and efficient online RLHF recipe for LLM instruct training. Our recipe is DPO - based and thus much cheaper and simpler to train and tune compared to PPO - based approaches. Unlike widely - used offline DPO, the online component of our approach effectively mitigates distribution shifts during policy optimization. For a detailed exposition, please refer to our accompanying technical report.

Chat Benchmarks

Academic Benchmarks

🔧 Technical Details

The online RLHF recipe is DPO - based. It is designed to be cheaper and simpler to train and tune than PPO - based approaches. The online component helps in reducing distribution shifts during policy optimization. For more details, refer to the accompanying technical report.

📄 License

The model is released under the Llama3 license.

⚠️ Important Note

Llama - 3 - 8B - SFR - Iterative - DPO - R is a research model developed as part of our RLHF initiative at Salesforce. While safety and ethical considerations are integral to our alignment process, there remains the possibility that the model could generate offensive or unethical content, particularly under adversarial conditions. We are committed to continuous improvement in our models to minimize such risks and encourage responsible usage.

💡 Usage Tip

Please cite our papers if you find our models are useful.

@misc{dong2024rlhf,
      title={RLHF Workflow: From Reward Modeling to Online RLHF}, 
      author={Hanze Dong* and Wei Xiong* and Bo Pang* and Haoxiang Wang* and Han Zhao and Yingbo Zhou and Nan Jiang and Doyen Sahoo and Caiming Xiong and Tong Zhang},
      year={2024},
      eprint={2405.07863},
      archivePrefix={arXiv},
      primaryClass={cs.LG}
}

@misc{xiong2024iterative,
      title={Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL - Constraint}, 
      author={Wei Xiong and Hanze Dong and Chenlu Ye and Ziqi Wang and Han Zhong and Heng Ji and Nan Jiang and Tong Zhang},
      year={2024},
      eprint={2312.11456},
      archivePrefix={arXiv},
      primaryClass={cs.LG}
}

We also recommend users to evaluate and address potential concerns related to accuracy, safety, and fairness before deploying this model. Refer to our standard AUP and [AI AUP](https://www.salesforce.com/content/dam/web/en_us/www/documents/legal/Agreements/policies/ai - acceptable - use - policy.pdf) for further guidance on use cases.