Fuyu 8b

🚀 Fuyu-8B Model Card

Fuyu-8B is a compact version of the multimodal model powering our product. It's now available on HuggingFace. Here's why we're excited about it:

1. It features a much simpler architecture and training process compared to other multimodal models. This simplicity makes it easier to understand, scale, and deploy.
2. It's built from the ground up for digital agents. It can handle arbitrary image resolutions, answer questions about graphs and diagrams, respond to UI - related queries, and perform fine - grained localization on screen images.
3. It's incredibly fast, capable of delivering responses for large images in under 100 milliseconds.
4. Despite being optimized for our specific use - case, it performs well on standard image understanding benchmarks like visual question - answering and natural - image - captioning.

Please note that the released model is a base model. You'll likely need to fine - tune it for specific use cases, such as verbose captioning or multimodal chat. In our experience, the model responds well to few - shot learning and fine - tuning for various use cases.

✨ Features

• Simplified Architecture: A vanilla decoder - only transformer without an image encoder, allowing support for arbitrary image resolutions.
• Fast Inference: Can generate responses for large images in less than 100 milliseconds.
• Versatile Application: Suitable for digital agents, with the ability to handle various image - related tasks.
• Good Benchmark Performance: Performs well on standard image understanding benchmarks.

📦 Installation

No installation steps are provided in the original document, so this section is skipped.

💻 Usage Examples

Basic Usage

from transformers import FuyuProcessor, FuyuForCausalLM
from PIL import Image
import requests

# load model and processor
model_id = "adept/fuyu-8b"
processor = FuyuProcessor.from_pretrained(model_id)
model = FuyuForCausalLM.from_pretrained(model_id, device_map="cuda:0")

# prepare inputs for the model
text_prompt = "Generate a coco-style caption.\n"
url = "https://huggingface.co/adept/fuyu-8b/resolve/main/bus.png"
image = Image.open(requests.get(url, stream=True).raw)

inputs = processor(text=text_prompt, images=image, return_tensors="pt").to("cuda:0")

# autoregressively generate text
generation_output = model.generate(**inputs, max_new_tokens=7)
generation_text = processor.batch_decode(generation_output[:, -7:], skip_special_tokens=True)
assert generation_text == ['A blue bus parked on the side of a road.']

Advanced Usage

# Question answering on natural images and charts/diagrams
text_prompt = "What color is the bus?\n"
url = "https://huggingface.co/adept/fuyu-8b/resolve/main/bus.png"
image = Image.open(requests.get(url, stream=True).raw)

inputs = processor(text=text_prompt, images=image, return_tensors="pt").to("cuda:0")

generation_output = model.generate(**inputs, max_new_tokens=6)
generation_text = processor.batch_decode(generation_output[:, -6:], skip_special_tokens=True)
assert generation_text == ["The bus is blue.\n"]


text_prompt = "What is the highest life expectancy at birth of male?\n"
url = "https://huggingface.co/adept/fuyu-8b/resolve/main/chart.png"
image = Image.open(requests.get(url, stream=True).raw)

model_inputs = processor(text=text_prompt, images=image, return_tensors="pt").to("cuda:0")

generation_output = model.generate(**model_inputs, max_new_tokens=16)
generation_text = processor.batch_decode(generation_output[:, -16:], skip_special_tokens=True)
assert generation_text == ["The life expectancy at birth of males in 2018 is 80.7.\n"]

💡 Usage Tip

For best performance, it's recommended to end questions with \n, as shown in the examples above!

📚 Documentation

Model

Fuyu-8B is a multi-modal text and image transformer trained by Adept AI.

Architecturally, Fuyu is a vanilla decoder - only transformer with no image encoder. Image patches are linearly projected into the first layer of the transformer, bypassing the embedding lookup. We treat the transformer decoder like an image transformer (albeit with no pooling and causal attention). See the diagram below for more details.

This simplification enables support for arbitrary image resolutions. We treat image token sequences like text token sequences, remove image - specific position embeddings, and feed in as many image tokens as needed in raster - scan order. A special image - newline character indicates line breaks. The model can use existing position embeddings to handle different image sizes, and we can use images of any size during training, eliminating the need for separate high and low - resolution training stages.

Model Description

Evaluation

Although not the main focus of this model, we evaluated it on standard image understanding benchmarks:

Uses

Direct Use

The model is for research purposes only. As this is a raw model release, we haven't added further fine - tuning, post - processing, or sampling strategies to control for undesirable outputs. You'll need to fine - tune the model for your use case.

Possible research areas and tasks include:

• Applications in computer control or digital agents.
• General research on multi - modal models.

Excluded uses are described below.

Out-of-Scope Use

The model wasn't trained to provide factual or accurate representations of people or events. So, using it to generate such content is beyond its capabilities.

Limitations and Bias

Limitations

• Faces and people in general may not be generated properly.

Bias

While these models have impressive capabilities, they can also reinforce or exacerbate social biases.

📄 License

The model is released under the [CC - BY - NC](https://creativecommons.org/licenses/by - nc/4.0/deed.en) license.