Reward Model Deberta V3 Large V2

🚀 Reward model trained from human feedback

This Reward Model (RM) is trained to predict the better-generated answer, as judged by humans, given a specific question. It offers significant value in multiple domains:

• Evaluating QA models
• Serving as a reward score in RLHF
• Detecting potentially toxic responses through ranking

All models are trained on the following datasets with a consistent split seed across them (if a validation split was unavailable):

• webgpt_comparisons
• summarize_from_feedback
• synthetic-instruct-gptj-pairwise
• anthropic_hh-rlhf

🚀 Quick Start

💻 Usage Examples

Basic Usage

from transformers import AutoModelForSequenceClassification, AutoTokenizer
reward_name = "OpenAssistant/reward-model-deberta-v3-large-v2"
rank_model, tokenizer = AutoModelForSequenceClassification.from_pretrained(reward_name), AutoTokenizer.from_pretrained(reward_name)
question, answer = "Explain nuclear fusion like I am five", "Nuclear fusion is the process by which two or more protons and neutrons combine to form a single nucleus. It is a very important process in the universe, as it is the source of energy for stars and galaxies. Nuclear fusion is also a key process in the production of energy for nuclear power plants."
inputs = tokenizer(question, answer, return_tensors='pt')
score = rank_model(**inputs).logits[0].cpu().detach()
print(score)

Advanced Usage

from transformers import AutoModelForSequenceClassification, AutoTokenizer
reward_name = "OpenAssistant/reward-model-deberta-v3-large-v2"
rank_model, tokenizer = AutoModelForSequenceClassification.from_pretrained(reward_name), AutoTokenizer.from_pretrained(reward_name)

question = "I just came out of from jail, any suggestion of my future?"
helpful = "It's great to hear that you have been released from jail."
bad = "Go back to jail you scum"

inputs = tokenizer(question, helpful, return_tensors='pt')
good_score = rank_model(**inputs).logits[0].cpu().detach()

inputs = tokenizer(question, bad, return_tensors='pt')
bad_score = rank_model(**inputs).logits[0].cpu().detach()
print(good_score > bad_score) # tensor([True])

📚 Documentation

Performance

The following table shows the validation split accuracy of different models:

It's likely that SytheticGPT has some surface patterns in the chosen-rejected pairs, making it relatively easy to distinguish between better answers.

📄 License

This project is licensed under the MIT license.

Other

Sincere thanks to stability.ai for their unwavering support in terms of A100 computational resources. Their contribution was crucial in ensuring the smooth completion of this research project.