Qwen2.5 Vl 7b Cam Motion Preview

🚀 视频文本处理模型

本项目基于预训练模型微调，在视频相机运动分类和视频文本检索任务上达到当前最优水平，可用于判断视频是否包含特定相机运动描述等场景。

🚀 快速开始

本模型的使用方法与 Qwen2.5-VL 模型相同，主要用于视频中的相机运动分类以及视频文本检索（在这两个任务中均为当前最优）。

✨ 主要特性

• 基于 Qwen/Qwen2.5-VL-7B-Instruct 模型在公开的高质量相机运动数据集上进行微调。
• 在相机运动分类和视频文本检索任务上达到当前最优水平。

📦 安装指南

文档未提及安装步骤，可参考 Qwen2.5-VL 模型的安装方法。

💻 使用示例

基础用法

# Import necessary libraries
from transformers import Qwen2_5_VLForConditionalGeneration, AutoProcessor
from qwen_vl_utils import process_vision_info
import torch

# Load the model
model = Qwen2_5_VLForConditionalGeneration.from_pretrained(
    "chancharikm/qwen2.5-vl-7b-cam-motion-preview", torch_dtype="auto", device_map="auto"
)
processor = AutoProcessor.from_pretrained("Qwen/Qwen2.5-VL-7B-Instruct")

# Prepare input data
video_path = "file:///path/to/video1.mp4"
text_description = "the camera tilting upward"
question = f"Does this video show \"{text_description}\"?"

# Format the input for the model
messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "video",
                "video": video_path,
                "fps": 8.0,  # Recommended FPS for optimal inference
            },
            {"type": "text", "text": question},
        ],
    }
]

text = processor.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True
)
image_inputs, video_inputs, video_kwargs = process_vision_info(messages, return_video_kwargs=True)
inputs = processor(
    text=[text],
    images=image_inputs,
    videos=video_inputs,
    padding=True,
    return_tensors="pt",
    **video_kwargs
)
inputs = inputs.to("cuda")

# Generate with score output
with torch.inference_mode():
    outputs = model.generate(
        **inputs,
        max_new_tokens=1,
        do_sample=False,  # Use greedy decoding to get reliable logprobs
        output_scores=True,
        return_dict_in_generate=True
    )

# Calculate probability of "Yes" response
scores = outputs.scores[0]
probs = torch.nn.functional.softmax(scores, dim=-1)
yes_token_id = processor.tokenizer.encode("Yes")[0]
score = probs[0, yes_token_id].item()

print(f"Video: {video_path}")
print(f"Description: '{text_description}'")
print(f"Score: {score:.4f}")

高级用法

# The model is trained on 8.0 FPS which we recommend for optimal inference

from transformers import Qwen2_5_VLForConditionalGeneration, AutoProcessor
from qwen_vl_utils import process_vision_info

# default: Load the model on the available device(s)
model = Qwen2_5_VLForConditionalGeneration.from_pretrained(
    "chancharikm/qwen2.5-vl-7b-cam-motion-preview", torch_dtype="auto", device_map="auto"
)

# We recommend enabling flash_attention_2 for better acceleration and memory saving, especially in multi-image and video scenarios.
# model = Qwen2_5_VLForConditionalGeneration.from_pretrained(
#     "chancharikm/qwen2.5-vl-7b-cam-motion-preview",
#     torch_dtype=torch.bfloat16,
#     attn_implementation="flash_attention_2",
#     device_map="auto",
# )

# default processor
processor = AutoProcessor.from_pretrained("Qwen/Qwen2.5-VL-7B-Instruct")

messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "video",
                "video": "file:///path/to/video1.mp4",
                "fps": 8.0,
            },
            {"type": "text", "text": "Describe the camera motion in this video."},
        ],
    }
]

text = processor.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True
)
image_inputs, video_inputs, video_kwargs = process_vision_info(messages, return_video_kwargs=True)
inputs = processor(
    text=[text],
    images=image_inputs,
    videos=video_inputs,
    fps=fps,
    padding=True,
    return_tensors="pt",
    **video_kwargs,
)
inputs = inputs.to("cuda")

# Inference
generated_ids = model.generate(**inputs, max_new_tokens=128)
generated_ids_trimmed = [
    out_ids[len(in_ids) :] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)
]
output_text = processor.batch_decode(
    generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False
)
print(output_text)

📚 详细文档

训练和评估数据

训练和评估数据可在我们的 仓库 中找到。

训练过程

我们使用 LLaMA-Factory 代码库对模型进行微调。如果需要复现我们的工作，请使用上述数据和以下超参数。

训练超参数

以下是训练过程中使用的超参数：

• 学习率：1e-05
• 训练批次大小：4
• 评估批次大小：1
• 随机种子：42
• 分布式类型：多 GPU
• 设备数量：8
• 梯度累积步数：8
• 总训练批次大小：256
• 总评估批次大小：8
• 优化器：使用 adamw_torch，β1=0.9，β2=0.999，ε=1e-08，无额外优化器参数
• 学习率调度器类型：余弦
• 学习率调度器热身比例：0.1
• 训练轮数：10.0

🔧 技术细节

本模型是 Qwen/Qwen2.5-VL-7B-Instruct 的微调版本，在当前公开的最高质量相机运动数据集上进行训练。该预览模型在相机运动分类和使用 VQAScore 进行相机运动字幕的视频文本检索任务中达到了当前最优水平。更多关于我们工作的信息可在我们的 CameraBench Github 页面上找到。

📄 许可证

许可证类型：其他

✏️ 引用

如果您发现本仓库对您的研究有用，请使用以下引用：

@article{lin2025camerabench,
  title={Towards Understanding Camera Motions in Any Video},
  author={Lin, Zhiqiu and Cen, Siyuan and Jiang, Daniel and Karhade, Jay and Wang, Hewei and Mitra, Chancharik and Ling, Tiffany and Huang, Yuhan and Liu, Sifan and Chen, Mingyu and Zawar, Rushikesh and Bai, Xue and Du, Yilun and Gan, Chuang and Ramanan, Deva},
  journal={arXiv preprint arXiv:2504.15376},
  year={2025},
}