UForm-Gen-Chat Open-Source Multimodal Dialogue Model - Free Implementation of Image Captioning and Visual Question Answering

Uform Gen Chat

Developed by unum-cloud

UForm-Gen-Chat is the fine-tuned multimodal conversational version of UForm-Gen, primarily used for image caption generation and visual question answering tasks.

Image-to-Text

Transformers

EnglishOpen Source License:Apache-2.0 #Lightweight Multimodal #Visual Question Answering #Image Caption Generation

Downloads 65

Release Time : 12/27/2023

Model Overview

UForm-Gen is a small generative vision-language model that includes a visual encoder and a language model fine-tuned on instruction datasets, suitable for image understanding and generation tasks.

Model Features

Multimodal Capability

Combines visual and language processing abilities to understand and generate content related to images

Lightweight

Compared to similar models, it has a smaller parameter size (1.5B), making it suitable for resource-limited environments

Conversation Optimized

Specifically fine-tuned for multimodal conversational scenarios

Model Capabilities

Image caption generation

Visual question answering

Multimodal conversation

Image content understanding

Use Cases

Content Understanding

Image caption generation

Generate natural language descriptions for input images

CLIPScore: 0.860 (long text), 0.858 (short text)

Visual question answering

Answer natural language questions about image content

Human-Computer Interaction

Multimodal conversation

Engage in natural language conversations based on image content

🚀 UForm

Pocket-Sized Multimodal AI for Content Understanding and Generation

🚀 Quick Start

UForm-Gen is a compact generative vision-language model mainly designed for Image Captioning and Visual Question Answering. It's composed of two parts:

UForm Vision Encoder
Sheared-LLaMA-1.3B manually tuned on the instructions dataset

The model was pre - trained on: MSCOCO, SBU Captions, Visual Genome, VQAv2, GQA and a few internal datasets. UForm - Gen - Chat is the SFT version of UForm - Gen for multimodal chat.

✨ Features

Designed for Image Captioning and Visual Question Answering.
Composed of UForm Vision Encoder and manually - tuned Sheared - LLaMA - 1.3B.
Pre - trained on multiple datasets.
Has an SFT version for multimodal chat.

📦 Installation

pip install uform

💻 Usage Examples

Basic Usage

For the CLI demo, run the following commands:

uform-chat --model unum-cloud/uform-gen-chat --image_path=zebra.jpg
uform-chat --model unum-cloud/uform-gen-chat --image_path=zebra.jpg --device="cuda:0" --fp16

Advanced Usage

If you want to use the model in your code:

from uform.gen_model import VLMForCausalLM, VLMProcessor

model = VLMForCausalLM.from_pretrained("unum-cloud/uform-gen-chat")
processor = VLMProcessor.from_pretrained("unum-cloud/uform-gen-chat")

prompt = "What do you see?"
image = Image.open("zebra.jpg")

inputs = processor(texts=[prompt], images=[image], return_tensors="pt")
with torch.inference_mode():
     output = model.generate(
        **inputs,
        do_sample=False,
        use_cache=True,
        max_new_tokens=128,
        eos_token_id=32001,
        pad_token_id=processor.tokenizer.pad_token_id
    )

prompt_len = inputs["input_ids"].shape[1]
decoded_text = processor.batch_decode(output[:, prompt_len:])[0]

📚 Documentation

Evaluation

For captioning evaluation, we measure CLIPScore and RefCLIPScore¹.

Property	Details
Model Type	UForm-Gen, a generative vision - language model
Training Data	MSCOCO, SBU Captions, Visual Genome, VQAv2, GQA and some internal datasets

Model	Size	Caption Length	CLIPScore	RefCLIPScore
`llava-hf/llava-1.5-7b-hf`	7B	Long	0.878	0.529
`llava-hf/llava-1.5-7b-hf`	7B	Short	0.886	0.531

`Salesforce/instructblip-vicuna-7b`	7B	Long	0.902	0.534
`Salesforce/instructblip-vicuna-7b`	7B	Short	0.848	0.523

`unum-cloud/uform-gen-chat`	1.5B	Long	0.860	0.525
`unum-cloud/uform-gen-chat`	1.5B	Short	0.858	0.525

¹ We used apple/DFN5B-CLIP-ViT-H-14-378 CLIP model.

📄 License

This project is licensed under the apache - 2.0 license.

Featured Recommended AI Models

Empowering the Future, Your AI Solution Knowledge Base

English 简体中文繁體中文にほんご