RM R1 DeepSeek Distilled Qwen 7B

🚀 RM - R1：奖励建模能否作为推理任务？

RM - R1 是一个用于 推理奖励模型（ReasRM）的训练框架，它通过先 大声思考 —— 生成结构化的评分标准或推理轨迹，然后给出偏好，来评判两个候选答案。与传统的标量或生成式奖励模型相比，RM - R1 在公共 RM 基准测试中平均表现出 最先进的性能，同时提供完全可解释的理由。

🚀 快速开始

资源链接

[🤗 模型与数据集] [📊 代码] [📖 论文]

✨ 主要特性

两阶段训练

1. 蒸馏：对约 8.7 K 条高质量推理轨迹（评分标准链）进行蒸馏。
2. 可验证奖励强化学习（RLVR）：在约 64 K 个偏好对上进行训练。

已发布的骨干模型

7 B / 14 B / 32 B Qwen - 2.5 - Instruct 变体 + DeepSeek 蒸馏检查点。

💡 预期用途

• 基于人类反馈的强化学习（RLHF）/ 基于AI反馈的强化学习（RLAIF）：可即插即用的奖励函数，用于策略优化。
• 自动评估：作为大语言模型评判器，用于开放域问答、聊天和推理任务。
• 研究：研究过程监督、思维链验证或评分标准生成。

💻 使用示例

基础用法

你可以使用以下示例代码来尝试该模型。完整的演示笔记本可在以下链接获取： 📎 官方演示链接

🧾 提示模板

REASONING_SINGLE_PROMPT_TEMPLATE = (
    "Please act as an impartial judge and evaluate the quality of the responses provided by two AI Chatbots to the Client question displayed below. \n\n"
    "[Client Question]\n{question}\n\n[The Start of Chatbot A's Response]\n{answer_a}\n[The End of Chatbot A's Response]\n\n"
    "[The Start of Chatbot B's Response]\n{answer_b}\n[The End of Chatbot B's Response]" + "\n\n"
    "Output your final verdict at last by strictly following this format: "
    "'<answer>[[A]]</answer>' if Chatbot A is better, or '<answer>[[B]]</answer>' if Chatbot B is better."
)

📦 推理代码

# Load the Model 
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "gaotang/RM-R1-DeepSeek-Distilled-Qwen-7B"
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype="auto",
    device_map="auto" # or specify the specific device map if needed
)
tokenizer = AutoTokenizer.from_pretrained(model_name)

# Single Turn Example - from Reward Bench 

prompt = "How do I detail a car?"
answer_a = "Detailing a car involves a thorough cleaning inside and out, as well as polishing and waxing to protect the vehicle's surfaces. Here's a step-by-step guide to detailing a car:\n\n**Exterior Detailing:**\n\n1. **Wash the Car:**\n   - Rinse the car with water to remove loose dirt.\n   - Use a car wash soap and microfiber wash mitt to clean the car from top to bottom.\n   - Clean the wheels and tires with a brush and a wheel cleaner.\n   - Rinse the car thoroughly to remove all soap.\n\n2. **Dry the Car:**\n   - Use a microfiber towel or a chamois to dry the car to prevent water spots.\n\n3. **Clay Bar Treatment:**\n   - Use a clay bar with a lubricant to remove embedded surface contaminants from the paint.\n\n4. **Polishing:**\n   - Apply car polish with a dual-action polisher or by hand to correct paint imperfections and create a smooth surface.\n\n5. **Waxing:**\n   - Apply a coat of wax or paint sealant to protect the paint and give it a glossy finish.\n\n6. **Windows and Mirrors:**\n   - Clean the windows and mirrors with a glass cleaner and a microfiber towel.\n\n7. **Tire and Trim Dressing:**\n   - Apply a tire dressing to the tires for a shiny finish.\n   - Use a trim restorer or protectant on plastic and rubber parts to prevent fading.\n\n**Interior Detailing:**\n\n1. **Remove Trash:**\n   - Clear out any trash and remove personal items from the car.\n\n2. **Vacuum:**\n   - Vacuum the seats, carpets, floor mats, and trunk.\n   - Use a brush attachment for the dashboard and door panels.\n\n3. **Shampoo Carpets and Upholstery:**\n   - Use a carpet cleaner and a brush to clean the carpets and upholstery.\n   - For leather interiors, use a leather cleaner and conditioner.\n\n4. **Clean Hard Surfaces:**\n   - Wipe down all hard surfaces (dashboard, center console, door panels, etc.) with a mild all-purpose cleaner and a microfiber cloth.\n\n5. **Windows and Mirrors:**\n   - Clean the interior side of windows and mirrors.\n\n6. **Air Vents and Crevices:**\n   - Use a detailing brush or compressed air to clean out air vents and hard-to-reach crevices.\n\n7. **Final Touches:**\n   - Apply a protectant to the dashboard and other plastic components.\n   - Replace air fresheners if needed.\n\n**Additional Tips:**\n\n- Work in the shade or a cool, well-ventilated garage to prevent products from drying too quickly and leaving residue.\n- Use separate buckets for washing and rinsing to avoid contaminating the clean water with dirt.\n- Always use gentle, non-abrasive materials and cleaners specifically designed for automotive use to avoid damaging surfaces.\n- Move in a systematic way to ensure you don't miss any spots.\n\nBy following these steps, you'll give your car a thorough clean that not only makes it look great but also helps in maintaining its value. Remember, regular detailing can prevent wear and tear and keep your car looking new for years to come." # Accepted
answer_b = "Detailing a car involves washing the exterior and interior of the car, as well as polishing and waxing the exterior. Interior detailing typically involves vacuuming, cleaning the upholstery and air vents, polishing the dashboard and console, and dusting. Polishing and waxing the exterior will depend on the condition of the paint, but typically involves applying a polish and wax to make it shine."  # Rejected

user_prompt_single = REASONING_SINGLE_PROMPT_TEMPLATE.format(
    question=prompt,
    answer_a=answer_a,
    answer_b=answer_b
) 

conversation = [
    {"role":"user", "content": user_prompt_single}
]

input_ids = tokenizer.apply_chat_template(
    conversation, 
    tokenize=True, 
    add_generation_prompt=True,
    return_tensors="pt").to(model.device)

generation = model.generate(
    input_ids=input_ids,
    max_new_tokens=8192,  # For optimal performance benchmarking, please set this to unlimited (e.g., 50000)
    do_sample=False,
)

completion = tokenizer.decode(
    generation[0][len(input_ids[0]):], 
    skip_special_tokens=True, 
    clean_up_tokenization_spaces=True
)

print(completion)

📚 详细文档

引用信息

@article{chen2025rm,
  title={RM-R1: Reward Modeling as Reasoning},
  author={Chen, Xiusi and Li, Gaotang and Wang, Ziqi and Jin, Bowen and Qian, Cheng and Wang, Yu and Wang, Hongru and Zhang, Yu and Zhang, Denghui and Zhang, Tong and others},
  journal={arXiv preprint arXiv:2505.02387},
  year={2025}
}

📄 许可证

本项目采用 MIT 许可证。