Llama-3-EvoVLM-JP-v2: An Open-Source Japanese Vision-Language Model - Support Interleaved Input of Text and Images

Llama 3 EvoVLM JP V2

Developed by SakanaAI

Llama-3-EvoVLM-JP-v2 is an experimental general-purpose Japanese vision-language model that supports interleaved input of text and images. This model was created using an evolutionary model fusion approach.

Image-to-Text

Transformers

Japanese#Japanese Multimodal #Interleaved Text-Image Processing #Evolutionary Model Fusion

Downloads 475

Release Time : 7/29/2024

Model Overview

Llama-3-EvoVLM-JP-v2 is a multimodal vision-language model supporting Japanese, capable of processing mixed inputs of text and images. By integrating the capabilities of multiple foundational models, it achieves visual language understanding and generation in Japanese environments.

Model Features

Multimodal Capability

Supports simultaneous processing of text and image inputs for visual language understanding

Japanese Optimization

Specially optimized for Japanese environments, suitable for Japanese users

Evolutionary Model Fusion

Utilizes an innovative model fusion method that combines the strengths of multiple excellent foundational models

Interleaved Input Support

Capable of handling complex inputs with interleaved text and images

Model Capabilities

Image Understanding

Japanese Text Generation

Visual Question Answering

Multimodal Reasoning

Image Caption Generation

Use Cases

Content Understanding

Japanese Image Captioning

Generates accurate textual descriptions for images in Japanese environments

Can produce image descriptions that conform to Japanese expression conventions

Visual Question Answering

Answers Japanese questions about image content

Capable of understanding image content and providing accurate answers in Japanese

Education

Japanese Learning Assistance

Helps Japanese learners through interactive use of images and text

Provides an intuitive Japanese learning experience

🐟 Llama-3-EvoVLM-JP-v2

An experimental general - purpose Japanese VLM that takes interleaved text and image as inputs.

🤗 Models | 📚 Paper | 📝 Blog | 🐦 Twitter

Llama-3-EvoVLM-JP-v2 is an experimental general - purpose Japanese Visual Language Model (VLM) that accepts interleaved text and image as inputs. This model was created using the Evolutionary Model Merge method. For more details, please refer to our report and blog. This model was produced by merging the following models, and we are grateful to the developers of the source models:

🚀 Quick Start

Use the following code to start using the model.

Click to expand

First, you need to install packages for inference using the Mantis. See here.

pip install git+https://github.com/TIGER-AI-Lab/Mantis.git

import requests
from PIL import Image

import torch
from mantis.models.conversation import Conversation, SeparatorStyle
from mantis.models.mllava import chat_mllava, LlavaForConditionalGeneration, MLlavaProcessor
from mantis.models.mllava.utils import conv_templates
from transformers import AutoTokenizer

# 1. Set the system prompt
conv_llama_3_elyza = Conversation(
    system="<|start_header_id|>system<|end_header_id|>\n\nあなたは誠実で優秀な日本人のアシスタントです。特に指示が無い場合は、常に日本語で回答してください。",
    roles=("user", "assistant"),
    messages=(),
    offset=0,
    sep_style=SeparatorStyle.LLAMA_3,
    sep="<|eot_id|>",
)
conv_templates["llama_3"] = conv_llama_3_elyza

# 2. Load model
device = "cuda" if torch.cuda.is_available() else "cpu"
model_id = "SakanaAI/Llama-3-EvoVLM-JP-v2"

processor = MLlavaProcessor.from_pretrained("TIGER-Lab/Mantis-8B-siglip-llama3")
processor.tokenizer.pad_token = processor.tokenizer.eos_token

model = LlavaForConditionalGeneration.from_pretrained(model_id, torch_dtype=torch.float16, device_map=device).eval()

# 3. Prepare a generate config
generation_kwargs = {
    "max_new_tokens": 128,
    "num_beams": 1,
    "do_sample": False,
    "no_repeat_ngram_size": 3,
}

# 4. Generate
text = "<image>の信号は何色ですか？"
url_list = [
    "https://images.unsplash.com/photo-1694831404826-3400c48c188d?q=80&w=2070&auto=format&fit=crop&ixlib=rb-4.0.3&ixid=M3wxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHx8fA%3D%3D",
    "https://images.unsplash.com/photo-1693240876439-473af88b4ed7?q=80&w=1974&auto=format&fit=crop&ixlib=rb-4.0.3&ixid=M3wxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHx8fA%3D%3D"
]
images = [
   Image.open(requests.get(url_list[0], stream=True).raw).convert("RGB")
]

response, history = chat_mllava(text, images, model, processor, **generation_kwargs)

print(response)
# 信号の色は、青色です。

# 5. Multi-turn conversation
text = "では、<image>の信号は？"
images += [
   Image.open(requests.get(url_list[1], stream=True).raw).convert("RGB")
]
response, history = chat_mllava(text, images, model, processor, history=history, **generation_kwargs)

print(response)
# 赤色

📚 Documentation

Model Details

Property	Details
Developed by	Sakana AI
Model Type	Autoregressive Language Model
Language(s)	Japanese
Optimization data	subsets of the Japanese Visual Genome VQA dataset and the translated ShareGPT4V
License	META LLAMA 3 COMMUNITY LICENSE
Paper	https://arxiv.org/abs/2403.13187
Blog	https://sakana.ai/evovlm-jp/

Uses

This model is provided for research and development purposes only and should be considered as an experimental prototype. It is not intended for commercial use or deployment in mission - critical environments. Use of this model is at the user's own risk, and its performance and outcomes are not guaranteed. Sakana AI shall not be liable for any direct, indirect, special, incidental, or consequential damages, or any loss arising from the use of this model, regardless of the results obtained. Users must fully understand the risks associated with the use of this model and use it at their own discretion.

🔗 Acknowledgement

We would like to thank the developers of the source models for their contributions and for making their work available.

📄 License

The model is under the META LLAMA 3 COMMUNITY LICENSE.

📖 Citation

@misc{Llama-3-EvoVLM-JP-v2, 
url    = {[https://huggingface.co/SakanaAI/Llama-3-EvoVLM-JP-v2](https://huggingface.co/SakanaAI/Llama-3-EvoVLM-JP-v2)}, 
title  = {Llama-3-EvoVLM-JP-v2}, 
author = {Yuichi, Inoue and Takuya, Akiba and Shing, Makoto}
}

@misc{akiba2024evomodelmerge,
      title         = {Evolutionary Optimization of Model Merging Recipes}, 
      author.       = {Takuya Akiba and Makoto Shing and Yujin Tang and Qi Sun and David Ha},
      year          = {2024},
      eprint        = {2403.13187},
      archivePrefix = {arXiv},
      primaryClass  = {cs.NE}
}

Featured Recommended AI Models

Empowering the Future, Your AI Solution Knowledge Base

English 简体中文繁體中文にほんご