Vsft Llava 1.5 7b Hf Trl

🚀 HuggingFaceH4/vsft-llava-1.5-7b-hf-trl 视觉语言模型

HuggingFaceH4/vsft-llava-1.5-7b-hf-trl 是一款视觉语言模型，它通过对 llava-hf/llava-1.5-7b-hf 模型进行 VSFT 处理得到，使用了来自 HuggingFaceH4/llava-instruct-mix-vsft 数据集的 260k 图像和对话对。

点击查看我们的 Spaces 演示！

🚀 快速开始

本模型支持多图像和多提示生成，即你可以在提示中传入多张图像。同时，请确保遵循正确的提示模板 (USER: xxx\nASSISTANT:)，并在需要查询图像的位置添加 <image> 标记。

✨ 主要特性

• 多模态处理：支持多图像和多提示生成。
• 遵循特定模板：需遵循 USER: xxx\nASSISTANT: 提示模板，并使用 <image> 标记查询图像。

📚 详细文档

模型详情

如何使用模型

💻 使用示例

基础用法

使用 pipeline：

from transformers import pipeline
from PIL import Image    
import requests

model_id = "HuggingFaceH4/vsft-llava-1.5-7b-hf-trl"
pipe = pipeline("image-to-text", model=model_id)
url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/ai2d-demo.jpg"

image = Image.open(requests.get(url, stream=True).raw)
prompt = "A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions. USER: <image>\nWhat does the label 15 represent? (1) lava (2) core (3) tunnel (4) ash cloud\nASSISTANT:"

outputs = pipe(image, prompt=prompt, generate_kwargs={"max_new_tokens": 200})
print(outputs)
>>> {"generated_text": "\nUSER: What does the label 15 represent? (1) lava (2) core (3) tunnel (4) ash cloud\nASSISTANT: Lava"}

高级用法

使用纯 transformers：

import requests
from PIL import Image

import torch
from transformers import AutoProcessor, LlavaForConditionalGeneration

model_id = "HuggingFaceH4/vsft-llava-1.5-7b-hf-trl"

prompt = "A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions. USER: <image>\nWhat are these?\nASSISTANT:"
image_file = "http://images.cocodataset.org/val2017/000000039769.jpg"

model = LlavaForConditionalGeneration.from_pretrained(
    model_id, 
    torch_dtype=torch.float16, 
    low_cpu_mem_usage=True, 
).to(0)

processor = AutoProcessor.from_pretrained(model_id)


raw_image = Image.open(requests.get(image_file, stream=True).raw)
inputs = processor(prompt, raw_image, return_tensors='pt').to(0, torch.float16)

output = model.generate(**inputs, max_new_tokens=200, do_sample=False)
print(processor.decode(output[0][2:], skip_special_tokens=True))

模型优化

4 位量化

通过 bitsandbytes 库进行 4 位量化：

model = LlavaForConditionalGeneration.from_pretrained(
    model_id, 
    torch_dtype=torch.float16, 
    low_cpu_mem_usage=True,
+   load_in_4bit=True
)

需先安装 bitsandbytes (pip install bitsandbytes)，并确保有支持 CUDA 的 GPU 设备。

使用 Flash-Attention 2

使用 Flash-Attention 2 进一步加速生成：

model = LlavaForConditionalGeneration.from_pretrained(
    model_id, 
    torch_dtype=torch.float16, 
    low_cpu_mem_usage=True,
+   use_flash_attention_2=True
).to(0)

需先安装 flash-attn，安装方法参考 Flash Attention 原仓库。

📄 许可证

Llama 2 遵循 LLAMA 2 社区许可证，版权归 Meta Platforms, Inc. 所有。

🔖 引用

@misc{vonwerra2022trl,
  author = {Leandro von Werra and Younes Belkada and Lewis Tunstall and Edward Beeching and Tristan Thrush and Nathan Lambert and Shengyi Huang},
  title = {TRL: Transformer Reinforcement Learning},
  year = {2020},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {\url{https://github.com/huggingface/trl}}
}