Asagi 8B

🚀 Asagi-8B: A Large-scale Japanese Vision & Language Model

This repository offers Asagi-8B, a large-scale Japanese Vision & Language Model (VLM). Trained on an extensive Japanese dataset with diverse data sources, it synthesizes a significant portion of training data using models like the Japanese large language model (CALM3-22B-Chat) and the English Vision & Language Model (Phi3.5-vision-instruct). Notably, it avoids using LLMs with output usage restrictions in their license terms (e.g., GPT-4) for data synthesis.

✨ Features

• Large-scale Japanese VLM: Asagi-8B is specifically designed for Japanese language processing in vision and language tasks.
• Diverse Training Data: Utilizes a wide range of data sources, including synthesized and translated datasets.
• Skipped Stage-1 Training: Unlike other models in the Asagi series, Asagi-8B skips stage-1 training.

📦 Installation

Requirements

transformers==4.45.1
accelerate==0.34.2
torch==2.4.0
torchvision==0.19.0

💻 Usage Examples

Basic Usage

import requests
import torch
import transformers
from PIL import Image
from transformers import AutoModel, AutoProcessor, GenerationConfig

transformers.set_seed(42)
model_path = "MIL-UT/Asagi-8B"
processor = AutoProcessor.from_pretrained(model_path)
model = AutoModel.from_pretrained(
    model_path, trust_remote_code=True,
    torch_dtype=torch.bfloat16,
    device_map="auto"
)

generation_config = GenerationConfig(
    do_sample=True,
    num_beams=5,
    max_new_tokens=256,
    temperature=0.7,
    repetition_penalty=1.5
)

prompt = ("The following is an instruction describing a task. Write a response that appropriately fulfills the request.\n\n"
            "### Instruction:\n<image>\nPlease look at this image and answer the following question in detail and specifically. Where was this photo taken? Also, please describe the content of the image in detail.\n\n### Response:\n")

# sample image
sample_image_url = "https://raw.githubusercontent.com/uehara-mech/uehara-mech.github.io/refs/heads/master/images/shibuya.jpg"
image = Image.open(requests.get(sample_image_url, stream=True).raw)

inputs = processor(
    text=prompt, images=image, return_tensors="pt"
)
inputs_text = processor.tokenizer(prompt, return_tensors="pt")
inputs['input_ids'] = inputs_text['input_ids']
inputs['attention_mask'] = inputs_text['attention_mask']
for k, v in inputs.items():
    if v.dtype == torch.float32:
        inputs[k] = v.to(model.dtype)
inputs = {k: inputs[k].to(model.device) for k in inputs if k != "token_type_ids"}

generate_ids = model.generate(
    **inputs,
    generation_config=generation_config
)
generated_text = processor.batch_decode(
    generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False
)[0]

# do not print the prompt
if "<image>" in prompt:
    prompt = prompt.replace("<image>", " ")
generated_text = generated_text.replace(prompt, "")

print(f"Generated text: {generated_text}")

# >>> Generated text:  This photo captures the night view of Ginza, Tokyo. Ginza is one of the representative bustling streets in Tokyo, lined with many high-rise buildings shining with the lights of neon signs and streetlights. Particularly striking is the "Ginza Crystal Building" towering in the center, with its characteristic clock tower being quite impressive. The cars and people moving between the buildings give a sense of the city's vitality.

Example

Note: This image was actually taken in Shibuya, not Ginza.

More Examples

📚 Documentation

Model Components

Training Details

Note: we skipped stage-1 training for Asagi-8B.

Note: ROIS (Ours) is a newly collected dataset crawled from the web specifically for this project. The dataset consists of image and raw text pairs, which are used to synthesize the training data.

Evaluation

We evaluated our model using Heron-Bench, JA-VLM-Bench-in-the-Wild, and JA-VG-VQA-500. We used the eval-mm library for this evaluation.

Here, models with "†" are not trained with GPT-generated data. Bold numbers indicate the best performance among all models, and underlined numbers indicate the best performance among models not trained with GPT-generated data.

🔧 Technical Details

The models released here are in the early stages of our research and development and have not been tuned to ensure outputs align with human intent and safety considerations.

📄 License

This project is licensed under the Apache-2.0 license.

Model Card Authors

Kohei Uehara