360VL 8B

🚀 360VL

360VL is developed based on the LLama3 language model. It's the industry's first open - source large multi - modal model based on LLama3 - 70B, and has a globally aware multi - branch projector architecture for better image understanding.

🚀 Quick Start

Here is a quick start example to use the 360VL model:

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
from PIL import Image

checkpoint = "qihoo360/360VL-8B"

model = AutoModelForCausalLM.from_pretrained(checkpoint, torch_dtype=torch.float16, device_map='auto', trust_remote_code=True).eval()
tokenizer = AutoTokenizer.from_pretrained(checkpoint, trust_remote_code=True)
vision_tower = model.get_vision_tower()
vision_tower.load_model()
vision_tower.to(device="cuda", dtype=torch.float16)
image_processor = vision_tower.image_processor
tokenizer.pad_token = tokenizer.eos_token


image = Image.open("docs/008.jpg").convert('RGB')
query = "Who is this cartoon character?"
terminators = [
    tokenizer.convert_tokens_to_ids("<|eot_id|>",)
]

inputs = model.build_conversation_input_ids(tokenizer, query=query, image=image, image_processor=image_processor)

input_ids = inputs["input_ids"].to(device='cuda', non_blocking=True)
images = inputs["image"].to(dtype=torch.float16, device='cuda', non_blocking=True)

output_ids = model.generate(
    input_ids,
    images=images,
    do_sample=False,
    eos_token_id=terminators,
    num_beams=1,
    max_new_tokens=512,
    use_cache=True)

input_token_len = input_ids.shape[1]
outputs = tokenizer.batch_decode(output_ids[:, input_token_len:], skip_special_tokens=True)[0]
outputs = outputs.strip()
print(outputs)

✨ Features

360VL offers the following features:

• Multi - round text - image conversations: 360VL can take both text and images as inputs and produce text outputs. Currently, it supports multi - round visual question answering with one image.
• Bilingual text support: 360VL supports conversations in both English and Chinese, including text recognition in images.
• Strong image comprehension: 360VL is adept at analyzing visuals, making it an efficient tool for tasks like extracting, organizing, and summarizing information from images.
• Fine - grained image resolution: 360VL supports image understanding at a higher resolution of 672×672.

📦 Model Zoo

360VL has released the following versions:

📊 Performance

📚 Documentation

Model type

360VL - 8B is an open - source chatbot trained by fine - tuning LLM on multimodal instruction - following data. It is an auto - regressive language model, based on the transformer architecture. Base LLM: [meta - llama/Meta - Llama - 3 - 8B - Instruct](https://huggingface.co/meta - llama/Meta - Llama - 3 - 8B - Instruct)

Model date

360VL - 8B was trained in April 2024.

📄 License

This project utilizes certain datasets and checkpoints that are subject to their respective original licenses. Users must comply with all terms and conditions of these original licenses. The content of this project itself is licensed under the [Apache license 2.0]

Where to send questions or comments about the model: https://github.com/360CVGroup/360VL

🔗 Related Projects

This work wouldn't be possible without the incredible open - source code of these projects. Huge thanks!

• [Meta Llama 3](https://github.com/meta - llama/llama3)
• [LLaVA: Large Language and Vision Assistant](https://github.com/haotian - liu/LLaVA)
• Honeybee: Locality - enhanced Projector for Multimodal LLM