Llava Llama 3 8b

🚀 LLaVA-llama-3-8B大语言模型

llava-llama-3-8b 是一个大型多模态模型（LMM），它基于 LLaVA-v1.5框架 进行训练。该模型以拥有80亿参数的 meta-llama/Meta-Llama-3-8B-Instruct 模型作为语言主干，并采用了基于CLIP的视觉编码器。

🚀 快速开始

本模型已针对多模态基准评估进行了微调，也可用作多模态聊天机器人。不过，请注意我们仅提供训练后的权重差异，不提供基础模型 meta-llama/Meta-Llama-3-8B-Instruct 的副本。使用这些权重需要单独下载基础模型。

✨ 主要特性

模型指标

模型详情

预期用途

影响因素

伦理考量

英特尔致力于尊重人权，避免对人权造成或促成不利影响。请参阅 英特尔全球人权原则。英特尔的产品和软件仅旨在用于不会对人权造成或促成不利影响的应用。

注意事项和建议

⚠️ 重要提示

用户（直接用户和下游用户）应了解该模型的风险、偏差和局限性。此模型尚未进行危害或偏差评估，不应用于可能造成危害的敏感应用。

📦 安装指南

由于仅提供权重差异，需要单独下载基础模型 meta-llama/Meta-Llama-3-8B-Instruct。

💻 使用示例

基础用法

# Copyright 2024 Intel Corporation
# SPDX-License-Identifier: Apache-2.0

import requests
import torch
from PIL import Image
from transformers import AutoProcessor, AutoModelForPreTraining
import transformers

def expand2square(pil_img, background_color):
    width, height = pil_img.size
    if width == height:
        return pil_img
    elif width > height:
        result = Image.new(pil_img.mode, (width, width), background_color)
        result.paste(pil_img, (0, (width - height) // 2))
        return result
    else:
        result = Image.new(pil_img.mode, (height, height), background_color)
        result.paste(pil_img, ((height - width) // 2, 0))
        return result

def add_model_a_to_b(model_a, model_b):
    state_dict_a = model_a.state_dict()
    state_dict_b = model_b.state_dict()

    # Ensure keys match before subtraction
    if set(state_dict_a.keys()) != set(state_dict_b.keys()):
        raise ValueError("Model state dicts do not have the same keys.")

    for key in state_dict_a:
        if state_dict_a[key].shape != state_dict_b[key].shape:
            raise ValueError(f"Shape mismatch for key '{key}': {state_dict_a[key].shape} vs {state_dict_b[key].shape}")
        # Subtract model_a's weights from model_b for the matching key
        state_dict_b[key] = state_dict_b[key] + state_dict_a[key]

    # Update model_b with the new weights
    model_b.load_state_dict(state_dict_b)

output_checkpoint = "" # set if you don't want to merge every time
hf_checkpoint = "Intel/llava-llama-3-8b"
device = "cuda" if torch.cuda.is_available() else "cpu"

processor = AutoProcessor.from_pretrained(hf_checkpoint)
model = AutoModelForPreTraining.from_pretrained(hf_checkpoint)
if model.language_model.model.embed_tokens.weight[-1].sum() == 0:
    print("adding llama3 weights")
    model_id = "meta-llama/Meta-Llama-3-8B-Instruct"
    pipeline = transformers.pipeline(
        "text-generation",
        model=model_id,
        model_kwargs={"torch_dtype": torch.bfloat16},
        device_map="cpu",
    )
    llama3 = pipeline.model
    add_model_a_to_b(llama3, model.language_model)
    if output_checkpoint:
        print("saving weights, so no adding is needed again")
        model.save_pretrained(output_checkpoint)

device = "cuda" if torch.cuda.is_available() else "cpu"
model.to(device)

prompt = processor.tokenizer.apply_chat_template(
    [{'role': 'user', 'content': "<image>\nWhat's the content of the image?"}],
    tokenize=False,
    add_generation_prompt=True
)

url = "https://www.ilankelman.org/stopsigns/australia.jpg"
image = Image.open(requests.get(url, stream=True).raw)

#original llava pads with mean, HF llava pads with zeros
image = expand2square(image, tuple(int(x*255) for x in processor.image_processor.image_mean)) 
inputs = processor(text=prompt, images=image, return_tensors="pt").to(device)
# Generate
generate_ids = model.generate(**inputs, max_length=30)
output = processor.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0]
print(output)

📚 详细文档

此模型卡由 Eduardo Alvarez 和上述作者创建。

📄 许可证

英特尔研究使用许可，所有使用代码遵循Apache 2.0许可。