UForm-Gen2-Qwen-500m Open-Source Vision-Language Model - Generate Image Descriptions for Free and Answer Visual Questions

Uform Gen2 Qwen 500m

Developed by unum-cloud

UForm-Gen is a small generative vision-language model primarily used for image caption generation and visual question answering.

Image-to-Text

Transformers

EnglishOpen Source License:Apache-2.0 #Lightweight multimodal #Image caption generation #Visual question answering

Downloads 17.98k

Release Time : 2/15/2024

Model Overview

UForm-Gen is a pocket-sized multimodal AI for content understanding and generation, featuring a CLIP-like ViT-H/14 visual encoder and Qwen1.5-0.5B-Chat language model.

Model Features

Lightweight design

Small generative vision-language model suitable for resource-constrained environments

Fast training

Training takes only one day using an DGX-H100 server with 8 H100 GPUs

Multimodal capability

Processes both visual and linguistic information for image understanding and generation

Model Capabilities

Image caption generation

Visual question answering

Multimodal dialogue

Content understanding

Content generation

Use Cases

Image understanding

Indoor scene description

Detailed description of indoor scenes

Generates text descriptions including furniture arrangement, decor style and other details

Animal behavior description

Brief description of animal behavior

Concise descriptions capturing animal postures and movement characteristics

Visual question answering

Image content Q&A

Answers specific questions about image content

Accurately answers questions about objects, scenes, etc. in images

🚀 UForm

Pocket-Sized Multimodal AI
For Content Understanding and Generation

UForm-Gen is a compact generative vision-language model tailored for Image Captioning and Visual Question Answering. This model comprises two key components:

CLIP-like ViT-H/14
Qwen1.5-0.5B-Chat

The model underwent pre-training on an internal image captioning dataset and fine-tuning on public instruction datasets such as SVIT, LVIS, and VQAs datasets. It was trained in one day on a DGX-H100 equipped with 8x H100 GPUs. Special thanks to Nebius.ai for providing the computing resources 🤗

✨ Features

Generate captions for images.
Answer questions about images.
Support multimodal chat.

📦 Installation

The installation steps are not provided in the original document, so this section is skipped.

💻 Usage Examples

Basic Usage

from transformers import AutoModel, AutoProcessor

model = AutoModel.from_pretrained("unum-cloud/uform-gen2-qwen-500m", trust_remote_code=True)
processor = AutoProcessor.from_pretrained("unum-cloud/uform-gen2-qwen-500m", trust_remote_code=True)

prompt = "Question or Instruction"
image = Image.open("image.jpg")

inputs = processor(text=[prompt], images=[image], return_tensors="pt")
with torch.inference_mode():
     output = model.generate(
        **inputs,
        do_sample=False,
        use_cache=True,
        max_new_tokens=256,
        eos_token_id=151645,
        pad_token_id=processor.tokenizer.pad_token_id
    )

prompt_len = inputs["input_ids"].shape[1]
decoded_text = processor.batch_decode(output[:, prompt_len:])[0]

You can check examples of different prompts in our demo space.

📚 Documentation

Model Information

Property	Details
Model Type	Generative vision-language model
Training Data	Internal image captioning dataset, public instruction datasets (SVIT, LVIS, VQAs datasets)

Evaluation

Model	LLM Size	SQA	MME	MMBench	Average¹
UForm-Gen2-Qwen-500m	0.5B	45.5	880.1	42.0	29.31
MobileVLM v2	1.4B	52.1	1302.8	57.7	36.81
LLaVA-Phi	2.7B	68.4	1335.1	59.8	42.95

¹MME scores were divided by 2000 before averaging.

📄 License

This project is licensed under the Apache-2.0 license.

Featured Recommended AI Models

Empowering the Future, Your AI Solution Knowledge Base

English 简体中文繁體中文にほんご