Decision Tree Reward Gemma 2 27B

🚀 通过决策树解读语言模型偏好

本项目旨在通过决策树的视角来解读语言模型的偏好，为语言模型的评估和优化提供新的思路和方法。

🚀 快速开始

在使用模型之前，请确保你已经安装了以下依赖：

• transformers==4.45.2
• torch>=2.5.0
• flash-attn>=2.6.3

注意：此代码需要使用具有NVIDIA Ampere架构或更新版本的GPU。

💻 使用示例

基础用法

from transformers import AutoModelForSequenceClassification
import torch
from transformers import AutoTokenizer
model_name = "Decision-Tree-Reward-Gemma-2-27B" # Another choice is "Decision-Tree-Reward-Llama-3.1-8B" 
repo_id = f"RLHFlow/{model_name}"
device = "cuda"
# Initialize the model and tokenizer
model = AutoModelForSequenceClassification.from_pretrained(repo_id, trust_remote_code=True, torch_dtype=torch.bfloat16, attn_implementation="flash_attention_2", device_map=device)
tokenizer = AutoTokenizer.from_pretrained(repo_id, use_fast=True)
# Load the decision tree
model.load_decision_tree(repo_id, filename="decision_tree.pkl")

# Prompt and response pairs
prompt = "Jane has 12 apples. She gives 4 apples to her friend Mark, then buys 1 more apple, and finally splits all her apples equally among herself and her 2 siblings. How many apples does each person get?"
response1 = "1. Jane starts with 12 apples and gives 4 to Mark. 12 - 4 = 8. Jane now has 8 apples.\n2. Jane buys 1 more apple. 8 + 1 = 9. Jane now has 9 apples.\n3. Jane splits the 9 apples equally among herself and her 2 siblings (3 people in total). 9 ÷ 3 = 3 apples each. Each person gets 3 apples."
response2 = "1. Jane starts with 12 apples and gives 4 to Mark. 12 - 4 = 8. Jane now has 8 apples.\n2. Jane buys 1 more apple. 8 + 1 = 9. Jane now has 9 apples.\n3. Jane splits the 9 apples equally among her 2 siblings (2 people in total). 9 ÷ 2 = 4.5 apples each. Each person gets 4 apples."

# Compare the two responses
output = model.compare(prompt, response1, response2, tokenizer, device)
print("Response 1 rewards")
print(dict(zip(output["attributes"], output["rewards"][0])))
# {'helpfulness': 3.7171721, 'correctness': 3.792478, 'coherence': 3.6601954, 'complexity': 0.8211964, 'verbosity': 1.8119512}
print("Response 2 rewards")
print(dict(zip(output["attributes"], output["rewards"][1])))
# {'helpfulness': -0.261065, 'correctness': -0.2378807, 'coherence': 2.4387608, 'complexity': 0.72620213, 'verbosity': 1.7181122}
print("Model preference")
print(output["preference"])
# 0

✨ 主要特性

• 模型排名展示：在RewardBench排行榜（2025年1月）中展示了不同模型的排名情况，包括整体得分以及在各个指标下的得分。
• 代码示例：提供了详细的使用代码示例，方便用户快速上手使用模型进行响应比较和奖励评估。

📚 详细文档

作者信息

• 作者：Min Li
• 博客：https://rlhflow.github.io/posts/2025-01-22-decision-tree-reward-model/

模型信息

• Decision-Tree-Reward-Gemma-2-27B
• Decision-Tree-Reward-Llama-3.1-8B

代码仓库

• 代码仓库地址：https://github.com/RLHFlow/RLHF-Reward-Modeling/tree/main/decision_tree

技术报告

• 技术报告状态：即将发布

RewardBench排行榜（2025年1月）

📄 许可证

⚠️ 重要提示

此模型是基于Skywork模型进行微调的，其使用需遵循以下许可协议：Skywork模型的社区使用需要遵循Skywork社区许可协议。Skywork模型支持商业使用。如果您计划将Skywork模型或其衍生模型用于商业目的，则必须遵守Skywork社区许可协议中的条款和条件。

📋 待办事项

• [x] 奖励模型使用代码
• [ ] 架构图