🚀 VideoLLaMA 2:推进视频大语言模型中的时空建模与音频理解
VideoLLaMA 2是一款多模态大语言模型,专注于视频领域,在时空建模和音频理解方面取得了显著进展,能有效处理视频问答、视频字幕生成等任务。
🚀 快速开始
本项目为视频大语言模型VideoLLaMA 2,如果你喜欢我们的项目,请在 Github 上给我们点个星 ⭐ 以获取最新更新。
📰 新闻动态
🌎 模型库
🚀 主要成果
多项选择视频问答与视频字幕生成

开放式视频问答

💻 使用示例
基础用法
import sys
sys.path.append('./')
from videollama2 import model_init, mm_infer
from videollama2.utils import disable_torch_init
def inference():
disable_torch_init()
modal = 'video'
modal_path = 'assets/cat_and_chicken.mp4'
instruct = '视频中有哪些动物,它们在做什么,视频给人的感觉如何?'
modal = 'image'
modal_path = 'assets/sora.png'
instruct = '图片中的女人穿着什么,她在做什么,图片给人的感觉如何?'
model_path = 'DAMO-NLP-SG/VideoLLaMA2-7B-16F'
model, processor, tokenizer = model_init(model_path)
output = mm_infer(processor[modal](modal_path), instruct, model=model, tokenizer=tokenizer, do_sample=False, modal=modal)
print(output)
if __name__ == "__main__":
inference()
📄 许可证
本项目采用Apache-2.0许可证。
引用信息
如果您发现VideoLLaMA对您的研究和应用有帮助,请使用以下BibTeX进行引用:
@article{damonlpsg2024videollama2,
title={VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs},
author={Cheng, Zesen and Leng, Sicong and Zhang, Hang and Xin, Yifei and Li, Xin and Chen, Guanzheng and Zhu, Yongxin and Zhang, Wenqi and Luo, Ziyang and Zhao, Deli and Bing, Lidong},
journal={arXiv preprint arXiv:2406.07476},
year={2024},
url = {https://arxiv.org/abs/2406.07476}
}
@article{damonlpsg2023videollama,
title = {Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding},
author = {Zhang, Hang and Li, Xin and Bing, Lidong},
journal = {arXiv preprint arXiv:2306.02858},
year = {2023},
url = {https://arxiv.org/abs/2306.02858}
}