Decision Tree Reward Gemma 2 27B

🚀 通過決策樹解讀語言模型偏好

本項目旨在通過決策樹的視角來解讀語言模型的偏好，為語言模型的評估和優化提供新的思路和方法。

🚀 快速開始

在使用模型之前，請確保你已經安裝了以下依賴：

• transformers==4.45.2
• torch>=2.5.0
• flash-attn>=2.6.3

注意：此代碼需要使用具有NVIDIA Ampere架構或更新版本的GPU。

💻 使用示例

基礎用法

from transformers import AutoModelForSequenceClassification
import torch
from transformers import AutoTokenizer
model_name = "Decision-Tree-Reward-Gemma-2-27B" # Another choice is "Decision-Tree-Reward-Llama-3.1-8B" 
repo_id = f"RLHFlow/{model_name}"
device = "cuda"
# Initialize the model and tokenizer
model = AutoModelForSequenceClassification.from_pretrained(repo_id, trust_remote_code=True, torch_dtype=torch.bfloat16, attn_implementation="flash_attention_2", device_map=device)
tokenizer = AutoTokenizer.from_pretrained(repo_id, use_fast=True)
# Load the decision tree
model.load_decision_tree(repo_id, filename="decision_tree.pkl")

# Prompt and response pairs
prompt = "Jane has 12 apples. She gives 4 apples to her friend Mark, then buys 1 more apple, and finally splits all her apples equally among herself and her 2 siblings. How many apples does each person get?"
response1 = "1. Jane starts with 12 apples and gives 4 to Mark. 12 - 4 = 8. Jane now has 8 apples.\n2. Jane buys 1 more apple. 8 + 1 = 9. Jane now has 9 apples.\n3. Jane splits the 9 apples equally among herself and her 2 siblings (3 people in total). 9 ÷ 3 = 3 apples each. Each person gets 3 apples."
response2 = "1. Jane starts with 12 apples and gives 4 to Mark. 12 - 4 = 8. Jane now has 8 apples.\n2. Jane buys 1 more apple. 8 + 1 = 9. Jane now has 9 apples.\n3. Jane splits the 9 apples equally among her 2 siblings (2 people in total). 9 ÷ 2 = 4.5 apples each. Each person gets 4 apples."

# Compare the two responses
output = model.compare(prompt, response1, response2, tokenizer, device)
print("Response 1 rewards")
print(dict(zip(output["attributes"], output["rewards"][0])))
# {'helpfulness': 3.7171721, 'correctness': 3.792478, 'coherence': 3.6601954, 'complexity': 0.8211964, 'verbosity': 1.8119512}
print("Response 2 rewards")
print(dict(zip(output["attributes"], output["rewards"][1])))
# {'helpfulness': -0.261065, 'correctness': -0.2378807, 'coherence': 2.4387608, 'complexity': 0.72620213, 'verbosity': 1.7181122}
print("Model preference")
print(output["preference"])
# 0

✨ 主要特性

• 模型排名展示：在RewardBench排行榜（2025年1月）中展示了不同模型的排名情況，包括整體得分以及在各個指標下的得分。
• 代碼示例：提供了詳細的使用代碼示例，方便用戶快速上手使用模型進行響應比較和獎勵評估。

📚 詳細文檔

作者信息

• 作者：Min Li
• 博客：https://rlhflow.github.io/posts/2025-01-22-decision-tree-reward-model/

模型信息

• Decision-Tree-Reward-Gemma-2-27B
• Decision-Tree-Reward-Llama-3.1-8B

代碼倉庫

• 代碼倉庫地址：https://github.com/RLHFlow/RLHF-Reward-Modeling/tree/main/decision_tree

技術報告

• 技術報告狀態：即將發佈

RewardBench排行榜（2025年1月）

📄 許可證

⚠️ 重要提示

此模型是基於Skywork模型進行微調的，其使用需遵循以下許可協議：Skywork模型的社區使用需要遵循Skywork社區許可協議。Skywork模型支持商業使用。如果您計劃將Skywork模型或其衍生模型用於商業目的，則必須遵守Skywork社區許可協議中的條款和條件。

📋 待辦事項

• [x] 獎勵模型使用代碼
• [ ] 架構圖