LLaVA-Next-Inst-It-Vicuna-7B开源模型 - 增强多模态实例理解，提升实际应用表现

首页

Llava Next Inst It Vicuna 7B

由 Inst-IT 开发

LLaVA-Next-Inst-It-Vicuna-7B 是一款在多模态实例级理解方面表现卓越的模型，通过显式视觉提示指令调优增强多模态实例理解。

文本到文本

Safetensors

英语开源协议:Apache-2.0 #实例级视觉理解 #多模态指令调优 #视频帧细粒度分析

下载量 14

发布时间 : 12/5/2024

模型简介

该模型基于 LLaVA-NeXT 架构，结合 Vicuna-7B 语言模型，专注于多模态实例级理解任务，支持图像和视频的细粒度分析。

模型特点

多模态实例级理解

通过显式视觉提示指令调优，增强对图像和视频中实例的细粒度理解能力。

支持 Set-of-Marks 视觉提示

可以利用 Set-of-Marks 视觉提示进行更精确的实例引用和分析。

视频帧时间戳引用

支持通过时间戳引用视频中的特定帧，实现时序感知的多模态理解。

模型能力

图像实例级描述

视频时序分析

多模态问答

细粒度视觉理解

开放式文本生成

使用案例

图像理解

图像实例描述

对图像中的特定实例进行详细描述，支持通过实例 ID 引用。

在 Inst-IT-Bench-I-OE 数据集上达到 68.6% 的准确率。

视频理解

视频时序分析

分析视频中特定时间点的内容变化，支持时间戳引用。

在 Inst-IT-Bench-V-OE 数据集上达到 49.3% 的准确率。

多模态问答

图像问答

回答关于图像内容的复杂问题，包括实例级细节。

在 GQA 数据集上达到 65.9% 的准确率。

🚀 LLaVA-Next-Inst-It-Vicuna-7B

LLaVA-Next-Inst-It-Vicuna-7B 是一个在实例级理解方面表现出色的多模态模型。该模型在论文 Inst-IT: Boosting Multimodal Instance Understanding via Explicit Visual Prompt Instruction Tuning 中被提出。

属性	详情
模型类型	clip-vit-large-patch14-336 + Vicuna-7B
初始化模型	LLaVA-NeXT
训练数据	LLaVA-NeXT-Data / Inst-IT-Dataset
精度	bfloat16

主页 | 代码 | 论文 | arXiv

🚀 快速开始

📦 安装指南

我们的代码基于 LLaVA-NeXT，在运行之前，请安装 LLaVA-NeXT 来准备环境：

pip install git+https://github.com/LLaVA-VL/LLaVA-NeXT.git

💻 使用示例

加载模型

from llava.model.builder import load_pretrained_model
from llava.constants import (
    DEFAULT_IMAGE_TOKEN,
    IMAGE_TOKEN_INDEX,
)
from llava.mm_utils import (
    KeywordsStoppingCriteria,
    get_model_name_from_path,
    tokenizer_image_token,
    process_images
)
from llava.conversation import SeparatorStyle, conv_templates

overwrite_config = {}
overwrite_config["mm_spatial_pool_stride"] = 2
overwrite_config["mm_spatial_pool_mode"] = 'bilinear'
overwrite_config["mm_pooling_position"] = 'after'
overwrite_config["mm_newline_position"] = 'no_token'

model_path = "Inst-IT/LLaVA-Next-Inst-It-Vicuna-7B"
model_name = get_model_name_from_path(model_path)

tokenizer, model, image_processor, max_length = load_pretrained_model(
            model_path=model_path, 
            model_base=None, 
            model_name=model_name,
            device_map="auto", 
            torch_dtype='bfloat16', 
            overwrite_config=overwrite_config,
            attn_implementation='sdpa')

图像推理

无 Set-of-Marks 的推理

我们的模型可以在没有 Set-of-Marks 视觉提示的图像上进行推理，在这种情况下，它可以像其基础模型 LLaVA-NeXT 一样使用。

import torch
import requests
from PIL import Image

img_url = "https://github.com/inst-it/inst-it/blob/main/assets/demo/image.jpg?raw=true"
image = Image.open(requests.get(img_url, stream=True).raw)
image_tensor = process_images([image], image_processor, model.config).bfloat16()
image_sizes = [image.size]

question = "Describe this image."
question = DEFAULT_IMAGE_TOKEN + "\n" + question

conv_template = 'vicuna_v1'
conv = conv_templates[conv_template].copy()
conv.append_message(conv.roles[0], question)
conv.append_message(conv.roles[1], None)
prompt = conv.get_prompt()

input_ids = tokenizer_image_token(prompt, tokenizer, IMAGE_TOKEN_INDEX, return_tensors="pt").unsqueeze(0).cuda()

pad_token_ids = tokenizer.pad_token_id if tokenizer.pad_token_id is not None else tokenizer.eos_token_id
attention_masks = input_ids.ne(pad_token_ids).long().cuda()

stop_str = conv.sep if conv.sep_style != SeparatorStyle.TWO else conv.sep2
keywords = [stop_str]
stopping_criteria = KeywordsStoppingCriteria(keywords, tokenizer, input_ids)

with torch.inference_mode():
    output_ids = model.generate(
        inputs=input_ids,
        images=image_tensor,
        attention_mask=attention_masks,
        modalities="image",
        image_sizes=image_sizes,
        use_cache=True,
        stopping_criteria=[stopping_criteria],
        max_new_tokens=4096
    )

pred = tokenizer.batch_decode(output_ids, skip_special_tokens=True)[0].strip()
print(pred)

有 Set-of-Marks 的推理

当提供 Set-of-Marks 视觉提示时，我们的模型可以进行更细粒度的理解。你可以使用实例的 ID 来引用你感兴趣的实例。与之前的推理代码相比，以下代码除了输入图像使用了 Set-of-Marks 进行视觉提示外，没有其他修改。请参考此链接了解如何为图像生成 Set-of-Marks。

import torch
import requests
from PIL import Image

img_url = "https://github.com/inst-it/inst-it/blob/main/assets/demo/image_som.jpg?raw=true"
image = Image.open(requests.get(img_url, stream=True).raw)
image_tensor = process_images([image], image_processor, model.config).bfloat16()
image_sizes = [image.size]

# You can use [id] to refer to the instances that you are interested in
question = "Describe [8] in detail."
question = DEFAULT_IMAGE_TOKEN + "\n" + question

conv_template = 'vicuna_v1'
conv = conv_templates[conv_template].copy()
conv.append_message(conv.roles[0], question)
conv.append_message(conv.roles[1], None)
prompt = conv.get_prompt()

input_ids = tokenizer_image_token(prompt, tokenizer, IMAGE_TOKEN_INDEX, return_tensors="pt").unsqueeze(0).cuda()

pad_token_ids = tokenizer.pad_token_id if tokenizer.pad_token_id is not None else tokenizer.eos_token_id
attention_masks = input_ids.ne(pad_token_ids).long().cuda()

stop_str = conv.sep if conv.sep_style != SeparatorStyle.TWO else conv.sep2
keywords = [stop_str]
stopping_criteria = KeywordsStoppingCriteria(keywords, tokenizer, input_ids)

with torch.inference_mode():
    output_ids = model.generate(
        inputs=input_ids,
        images=image_tensor,
        attention_mask=attention_masks,
        modalities="image",
        image_sizes=image_sizes,
        use_cache=True,
        stopping_criteria=[stopping_criteria],
        max_new_tokens=4096
    )

pred = tokenizer.batch_decode(output_ids, skip_special_tokens=True)[0].strip()
print(pred)

视频推理

对于视频，我们将每一帧组织成一个列表。你可以使用格式 <t> 来引用特定的时间戳（例如 <1>）。

无 Set-of-Marks 的推理

我们的模型可以在没有 Set-of-Marks 视觉提示的视频上进行推理，在这种情况下，它可以像其基础模型 LLaVA-NeXT 一样使用。

import torch
import requests
from PIL import Image

frame_urls = [
    "https://github.com/inst-it/inst-it/blob/main/assets/demo/frame_1.jpg?raw=true",
    "https://github.com/inst-it/inst-it/blob/main/assets/demo/frame_2.jpg?raw=true",
    "https://github.com/inst-it/inst-it/blob/main/assets/demo/frame_3.jpg?raw=true",
    "https://github.com/inst-it/inst-it/blob/main/assets/demo/frame_4.jpg?raw=true",
    "https://github.com/inst-it/inst-it/blob/main/assets/demo/frame_5.jpg?raw=true",
    "https://github.com/inst-it/inst-it/blob/main/assets/demo/frame_6.jpg?raw=true",
    "https://github.com/inst-it/inst-it/blob/main/assets/demo/frame_7.jpg?raw=true",
    "https://github.com/inst-it/inst-it/blob/main/assets/demo/frame_8.jpg?raw=true"
]
video = [Image.open(requests.get(frame_url, stream=True).raw) for frame_url in frame_urls]
video = image_processor.preprocess(video, return_tensors="pt")["pixel_values"].cuda()
video = video.bfloat16()
videos = [video]

question = "Describe the video."  # overall video caption
question = "What happens at frame <1>?"  # caption a specific moment
question = DEFAULT_IMAGE_TOKEN + "\n" + question

conv_template = 'vicuna_v1'
conv = conv_templates[conv_template].copy()
conv.append_message(conv.roles[0], question)
conv.append_message(conv.roles[1], None)
prompt = conv.get_prompt()

input_ids = tokenizer_image_token(prompt, tokenizer, IMAGE_TOKEN_INDEX, return_tensors="pt").unsqueeze(0).cuda()

pad_token_ids = tokenizer.pad_token_id if tokenizer.pad_token_id is not None else tokenizer.eos_token_id
attention_masks = input_ids.ne(pad_token_ids).long().cuda()

stop_str = conv.sep if conv.sep_style != SeparatorStyle.TWO else conv.sep2
keywords = [stop_str]
stopping_criteria = KeywordsStoppingCriteria(keywords, tokenizer, input_ids)

with torch.inference_mode():
    output_ids = model.generate(
        inputs=input_ids,
        images=videos,
        attention_mask=attention_masks,
        modalities="video",
        use_cache=True,
        stopping_criteria=[stopping_criteria],
        max_new_tokens=4096
    )

pred = tokenizer.batch_decode(output_ids, skip_special_tokens=True)[0].strip()
print(pred)

有 Set-of-Marks 的推理

当提供 Set-of-Marks 视觉提示时，我们的模型可以进行更细粒度的理解。你可以使用实例的 ID 来引用你感兴趣的实例。与之前的推理代码相比，以下代码除了输入视频使用了 Set-of-Marks 进行视觉提示外，没有其他修改。请参考 SAM2 和 SoM 了解如何为视频生成 Set-of-Marks。

import torch
import requests
from PIL import Image

frame_urls = [
    "https://github.com/inst-it/inst-it/blob/main/assets/demo/som_frame_1.jpg?raw=true",
    "https://github.com/inst-it/inst-it/blob/main/assets/demo/som_frame_2.jpg?raw=true",
    "https://github.com/inst-it/inst-it/blob/main/assets/demo/som_frame_3.jpg?raw=true",
    "https://github.com/inst-it/inst-it/blob/main/assets/demo/som_frame_4.jpg?raw=true",
    "https://github.com/inst-it/inst-it/blob/main/assets/demo/som_frame_5.jpg?raw=true",
    "https://github.com/inst-it/inst-it/blob/main/assets/demo/som_frame_6.jpg?raw=true",
    "https://github.com/inst-it/inst-it/blob/main/assets/demo/som_frame_7.jpg?raw=true",
    "https://github.com/inst-it/inst-it/blob/main/assets/demo/som_frame_8.jpg?raw=true"
]
video = [Image.open(requests.get(frame_url, stream=True).raw) for frame_url in frame_urls]
video = image_processor.preprocess(video, return_tensors="pt")["pixel_values"].cuda()
video = video.bfloat16()
videos = [video]

# You can use [id] to refer to the instances that you are interested in
question = "Is [3] visible at <1>?"
question = DEFAULT_IMAGE_TOKEN + "\n" + question

conv_template = 'vicuna_v1'
conv = conv_templates[conv_template].copy()
conv.append_message(conv.roles[0], question)
conv.append_message(conv.roles[1], None)
prompt = conv.get_prompt()

input_ids = tokenizer_image_token(prompt, tokenizer, IMAGE_TOKEN_INDEX, return_tensors="pt").unsqueeze(0).cuda()

pad_token_ids = tokenizer.pad_token_id if tokenizer.pad_token_id is not None else tokenizer.eos_token_id
attention_masks = input_ids.ne(pad_token_ids).long().cuda()

stop_str = conv.sep if conv.sep_style != SeparatorStyle.TWO else conv.sep2
keywords = [stop_str]
stopping_criteria = KeywordsStoppingCriteria(keywords, tokenizer, input_ids)

with torch.inference_mode():
    output_ids = model.generate(
        inputs=input_ids,
        images=videos,
        attention_mask=attention_masks,
        modalities="video",
        use_cache=True,
        stopping_criteria=[stopping_criteria],
        max_new_tokens=4096
    )

pred = tokenizer.batch_decode(output_ids, skip_special_tokens=True)[0].strip()
print(pred)

📞 联系我们

如果您有任何问题或建议，请随时与我们联系：

邮箱（彭武健）：wjpeng24@m.fudan.edu.cn
邮箱（孟令晨）：lcmeng20@fudan.edu.cn

📄 许可证

本项目采用 Apache-2.0 许可证。

📚 引用

@article{peng2024inst,
  title={Inst-IT: Boosting Multimodal Instance Understanding via Explicit Visual Prompt Instruction Tuning},
  author={Peng, Wujian and Meng, Lingchen and Chen, Yitong and Xie, Yiweng and Liu, Yang and Gui, Tao and Xu, Hang and Qiu, Xipeng and Wu, Zuxuan and Jiang, Yu-Gang},
  journal={arXiv preprint arXiv:2412.03565},
  year={2024}
}