Gpt2 Large Harmless Reward Model

🚀 GPT2大模型：有害响应检测与RLHF

本项目是在Anthropic/hh - rlhf无害数据集上训练的GPT2大模型，专门用于有害响应检测或基于人类反馈的强化学习（RLHF）。该模型在测试集上达到了0.73698的准确率，几乎可与其他更大规模的模型相媲美。

🚀 快速开始

使用示例

基础用法

import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification

rm_tokenizer = AutoTokenizer.from_pretrained('Ray2333/gpt2-large-harmless-reward_model')
reward_model = AutoModelForSequenceClassification.from_pretrained(
                'Ray2333/gpt2-large-harmless-reward_model',
                num_labels=1, torch_dtype=torch.bfloat16,
                device_map=0,
                )
q, a = "\n\nHuman: I just came out of from jail, any suggestion of my future? \n\nAssistant:", "Go back to jail you scum"
inputs = rm_tokenizer(q, a, return_tensors='pt', truncation=True)
with torch.no_grad():
  reward = reward_model(**(inputs.to(0))).logits[0].cpu().detach().item()

📚 详细文档

注意事项

⚠️ 重要提示

1. 推理时请使用Anthropic/hh - rlhf数据集的格式。
2. 此奖励模型与其他在完整Anthropic/hh - rlhf数据集上训练的开源奖励模型不同。

引用信息

本奖励模型用于ICML 2024的Rewards - in - context项目中的多目标对齐（特别是“无害”和“有用”对齐）。

@article{yang2024rewards,
  title={Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustment},
  author={Yang, Rui and Pan, Xiaoman and Luo, Feng and Qiu, Shuang and Zhong, Han and Yu, Dong and Chen, Jianshu},
  journal={International Conference on Machine Learning},
  year={2024}
}

📄 许可证

本项目采用MIT许可证。

📦 数据集与指标