Gpt2 Large Harmless Reward Model

🚀 GPT2大模型：有害響應檢測與RLHF

本項目是在Anthropic/hh - rlhf無害數據集上訓練的GPT2大模型，專門用於有害響應檢測或基於人類反饋的強化學習（RLHF）。該模型在測試集上達到了0.73698的準確率，幾乎可與其他更大規模的模型相媲美。

🚀 快速開始

使用示例

基礎用法

import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification

rm_tokenizer = AutoTokenizer.from_pretrained('Ray2333/gpt2-large-harmless-reward_model')
reward_model = AutoModelForSequenceClassification.from_pretrained(
                'Ray2333/gpt2-large-harmless-reward_model',
                num_labels=1, torch_dtype=torch.bfloat16,
                device_map=0,
                )
q, a = "\n\nHuman: I just came out of from jail, any suggestion of my future? \n\nAssistant:", "Go back to jail you scum"
inputs = rm_tokenizer(q, a, return_tensors='pt', truncation=True)
with torch.no_grad():
  reward = reward_model(**(inputs.to(0))).logits[0].cpu().detach().item()

📚 詳細文檔

注意事項

⚠️ 重要提示

1. 推理時請使用Anthropic/hh - rlhf數據集的格式。
2. 此獎勵模型與其他在完整Anthropic/hh - rlhf數據集上訓練的開源獎勵模型不同。

引用信息

本獎勵模型用於ICML 2024的Rewards - in - context項目中的多目標對齊（特別是“無害”和“有用”對齊）。

@article{yang2024rewards,
  title={Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustment},
  author={Yang, Rui and Pan, Xiaoman and Luo, Feng and Qiu, Shuang and Zhong, Han and Yu, Dong and Chen, Jianshu},
  journal={International Conference on Machine Learning},
  year={2024}
}

📄 許可證

本項目採用MIT許可證。

📦 數據集與指標