Llama 3 OffsetBias RM 8B

🚀 Llama-3-OffsetBias-RM-8B

A reward model trained on the OffsetBias dataset, designed to be more robust against various evaluation biases.

🚀 Quick Start

Llama-3-OffsetBias-RM-8B is a reward model trained on the OffsetBias dataset. It aims to be more robust against various evaluation biases commonly found in evaluation models. The model is introduced in the paper "OffsetBias: Leveraging Debiased Data for Tuning Evaluators".

✨ Features

• Trained on the OffsetBias dataset to enhance robustness against evaluation biases.
• Built upon Meta Llama 3 architecture.

📦 Installation

No specific installation steps are provided in the original document.

💻 Usage Examples

Basic Usage

from transformers import AutoTokenizer, pipeline
import torch

model_name = "NCSOFT/Llama-3-OffsetBias-RM-8B"
rm_tokenizer = AutoTokenizer.from_pretrained(model_name)
rm_pipe = pipeline(
    "sentiment-analysis",
    model=model_name,
    device="auto",
    tokenizer=rm_tokenizer,
    model_kwargs={"torch_dtype": torch.bfloat16}
)

pipe_kwargs = {
    "return_all_scores": True,
    "function_to_apply": "none",
    "batch_size": 1
}

chat = [
 {"role": "user", "content": "Hello, how are you?"},
 {"role": "assistant", "content": "I'm doing great. How can I help you today?"},
 {"role": "user", "content": "I'd like to show off how chat templating works!"},
]

test_texts = [rm_tokenizer.apply_chat_template(chat, tokenize=False, add_generation_prompt=False).replace(rm_tokenizer.bos_token, "")]
pipe_outputs = rm_pipe(test_texts, **pipe_kwargs)
rewards = [output[0]["score"] for output in pipe_outputs]

📚 Documentation

Model Details

Model Description

Llama-3-OffsetBias-RM-8B uses sfairXC/FsfairX-LLaMA3-RM-v0.1 as the base model, which is built with Meta Llama 3. An intermediate reward model is trained from Llama-3-8B-Instruct using a subset of the dataset used in the training of the FsfairX-LLaMA3-RM model, combined with the NCSOFT/offsetbias dataset. The intermediate model is then merged with the FsfairX-LLaMA3-RM model to create Llama-3-OffsetBias-RM-8B.

Model Sources

• 💻 Repository: https://github.com/ncsoft/offsetbias
• 📜 Paper: OffsetBias: Leveraging Debiased Data for Tuning Evaluators
• 🤗 Dataset: https://huggingface.co/datasets/NCSOFT/offsetbias

Uses

The model can be used directly as shown in the code example above for sentiment analysis tasks.

Evaluation

RewardBench Result

EvalBiasBench Result

📄 License

The model is released under the META LLAMA 3 COMMUNITY LICENSE AGREEMENT.

📚 Citation

@misc{park2024offsetbias,
      title={OffsetBias: Leveraging Debiased Data for Tuning Evaluators},
      author={Junsoo Park and Seungyeon Jwa and Meiying Ren and Daeyoung Kim and Sanghyuk Choi},
      year={2024},
      eprint={2407.06551},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}