Skywork VL Reward 7B

🚀 Skywork-VL-Reward

Skywork-VL-Reward是一款开源的7B多模态奖励模型，基于Qwen2.5-VL-7B-Instruct架构，通过添加价值头结构进行训练。该模型在VL-RewardBench和RewardBench等评估基准中取得了优异成绩，为多模态强化学习领域注入了新的活力。

🚀 快速开始

环境搭建

conda create -n vl-reward python=3.11
conda activate vl-reward
bash setup.sh

运行推理代码

import torch
from transformers import AutoProcessor, Qwen2_5_VLForConditionalGeneration
from trl import AutoModelForCausalLMWithValueHead
from qwen_vl_utils import process_vision_info
from transformers.utils import cached_file
from safetensors import safe_open


processor = AutoProcessor.from_pretrained("Skywork/Skywork-VL-Reward-7B")
# The default range for the number of visual tokens per image in the model is 4-16384.
# You can set min_pixels and max_pixels according to your needs, such as a token range of 256-1280, to balance performance and cost.
# min_pixels = 256*28*28
# max_pixels = 1280*28*28
# processor = AutoProcessor.from_pretrained("Skywork/Skywork-VL-Reward-7B", min_pixels=min_pixels, max_pixels=max_pixels)

model = Qwen2_5_VLForConditionalGeneration.from_pretrained(
    "Skywork/Skywork-VL-Reward-7B",
    device_map="auto",
    torch_dtype=torch.bfloat16,
)
# We recommend enabling flash_attention_2 for better acceleration and memory saving
# pip install flash-attn --no-build-isolation
#
# model = Qwen2_5_VLForConditionalGeneration.from_pretrained(
#     "Skywork/Skywork-VL-Reward-7B",
#     device_map="auto",
#     torch_dtype=torch.bfloat16,
#     attn_implementation="flash_attention_2",
# )

model = AutoModelForCausalLMWithValueHead.from_pretrained(model)
vhead_file = cached_file(
    path_or_repo_id="Skywork/Skywork-VL-Reward-7B", filename="value_head.safetensors"
)
with safe_open(vhead_file, framework="pt", device="cpu") as f:
    vhead_params = {key: f.get_tensor(key) for key in f.keys()}
model.load_state_dict(vhead_params, strict=False)
model.requires_grad_(False)
model.eval()

# score: 23.89
# if you use flash_attention_2 the score will be 23.76
demo_image = "demo.jpg"
demo_question = "Hint: Please answer the question and provide the correct option letter, e.g., A, B, C, D, at the end.\nQuestion: Is Purple the highest value?\nChoices:\n(A) no\n(B) yes"
demo_answer = "The answer is: B"

messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "image",
                "image": demo_image,
            },
            {
                "type": "text",
                "text": demo_question,
            },
        ],
    },
    {
        "role": "assistant",
        "content": demo_answer,
    },
]
text = processor.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=False
)
image_inputs, video_inputs = process_vision_info(messages)
inputs = processor(
    text=[text],
    images=image_inputs,
    videos=video_inputs,
    padding=True,
    return_tensors="pt",
)
inputs = inputs.to("cuda")
values = model(**inputs, return_dict=True, use_cache=False)[-1]
scores = values.gather(
    dim=-1, index=(inputs["attention_mask"].sum(dim=-1, keepdim=True) - 1)
)
score = scores[0].item()
print("Reward Score is: ", score)

✨ 主要特性

• 开源多模态奖励模型：Skywork-VL-Reward是一款开源的7B多模态奖励模型，为多模态强化学习领域提供了新的解决方案。
• 基于Qwen2.5-VL-7B-Instruct架构：该模型基于Qwen2.5-VL-7B-Instruct架构，通过添加价值头结构进行训练，提高了模型的性能。
• 优异的评估成绩：在VL-RewardBench和RewardBench等评估基准中，Skywork-VL-Reward取得了优异的成绩，证明了其有效性和优越性。

📚 详细文档

🔥 最新消息

• 2025年5月12日：我们的技术报告已在arXiv上发布，欢迎引用：Skywork-VL Reward: An Effective Reward Model for Multimodal Understanding and Reasoning
• 2025年4月24日：我们发布了Skywork-VL-Reward-7B，这是一款在VLRewardBench上表现出色的多模态奖励模型，并在R1V GitHub仓库中发布了技术报告。

技术报告

Skywork-VL Reward: An Effective Reward Model for Multimodal Understanding and Reasoning

评估结果

VL-RewardBench

RewardBench

📄 许可证

本项目采用MIT许可证。

📝 引用

如果您在研究中使用了本项目的成果，请引用以下文献：

@misc{wang2025skyworkvlrewardeffectivereward,
      title={Skywork-VL Reward: An Effective Reward Model for Multimodal Understanding and Reasoning}, 
      author={Xiaokun Wang and Chris and Jiangbo Pei and Wei Shen and Yi Peng and Yunzhuo Hao and Weijie Qiu and Ai Jian and Tianyidan Xie and Xuchen Song and Yang Liu and Yahui Zhou},
      year={2025},
      eprint={2505.07263},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2505.07263}, 
}