Idefics2 8b Chatty

🚀 Idefics2

Idefics2 is an open multimodal model that accepts arbitrary sequences of image and text inputs and produces text outputs. It can answer questions about images, describe visual content, create stories based on multiple images, or act as a pure language model without visual inputs. It significantly improves upon Idefics1, enhancing capabilities in OCR, document understanding, and visual reasoning.

🚀 Quick Start

This section shows code snippets for generation with idefics2-8b-base and idefics2-8b. The codes only differ in input formatting. First, let's define some common imports and inputs.

import requests
import torch
from PIL import Image
from io import BytesIO

from transformers import AutoProcessor, AutoModelForVision2Seq
from transformers.image_utils import load_image

DEVICE = "cuda:0"

# Note that passing the image urls (instead of the actual pil images) to the processor is also possible
image1 = load_image("https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg")
image2 = load_image("https://cdn.britannica.com/59/94459-050-DBA42467/Skyline-Chicago.jpg")
image3 = load_image("https://cdn.britannica.com/68/170868-050-8DDE8263/Golden-Gate-Bridge-San-Francisco.jpg")

Basic Usage

For idefics2-8b-base

processor = AutoProcessor.from_pretrained("HuggingFaceM4/idefics2-8b-base")
model = AutoModelForVision2Seq.from_pretrained(
    "HuggingFaceM4/idefics2-8b-base",
).to(DEVICE)

# Create inputs
prompts = [
  "<image>In this image, we can see the city of New York, and more specifically the Statue of Liberty.<image>In this image,",
  "In which city is that bridge located?<image>",
]
images = [[image1, image2], [image3]]
inputs = processor(text=prompts, images=images, padding=True, return_tensors="pt")
inputs = {k: v.to(DEVICE) for k, v in inputs.items()}


# Generate
generated_ids = model.generate(**inputs, max_new_tokens=500)
generated_texts = processor.batch_decode(generated_ids, skip_special_tokens=True)

print(generated_texts)
# ['In this image, we can see the city of New York, and more specifically the Statue of Liberty. In this image, we can see the city of Chicago, and more specifically the skyscrapers of the city.', 'In which city is that bridge located? The Golden Gate Bridge is a suspension bridge spanning the Golden Gate, the one-mile-wide (1.6 km) strait connecting San Francisco Bay and the Pacific Ocean. The structure links the American city of San Francisco, Calif

✨ Features

• Accepts arbitrary sequences of image and text inputs and produces text outputs.
• Can answer questions about images, describe visual content, create stories based on multiple images, or act as a pure language model without visual inputs.
• Significantly improves upon Idefics1, enhancing capabilities in OCR, document understanding, and visual reasoning.

📦 Installation

No installation steps were provided in the original document, so this section is skipped.

📚 Documentation

Model Summary

Uses

idefics2-8b-base and idefics2-8b can be used for inference on multimodal (image + text) tasks where the input consists of a text query and one (or multiple) images. Text and images can be interleaved arbitrarily. This includes image captioning, visual question answering, etc. These models do not support image generation.

For optimal results, it is recommended to fine-tune idefics2-8b on specific use-cases and data. The instruction-fine-tuned model (idefics2-8b) is better at following user instructions and should be preferred for out-of-the-box use or as a starting point for fine-tuning.

idefics2-8b usually generates very short answers. For long generations, use idefics2-8b-chatty, which was further fine-tuned on long conversations.

As a starting point, fine-tuning codes are provided that can be adapted for specific scenarios:

• With the TRL library: Script
• With the Hugging Face Trainer: Tutorial notebook

Technical summary

Idefics2 shows strong performance for a model of its size (8B parameters) compared to other open multimodal models and is often competitive with closed-source systems. It serves as a strong foundation for various use-case specific fine-tunings.

Idefics2 introduces several carefully abalated improvements over Idefics1:

• Manipulates images in their native resolutions (up to 980 x 980) and native aspect ratios by following the NaViT strategy, avoiding the need to resize images to fixed-size squares. Optionally, it allows sub-image splitting and passing images of very large resolution following the SPHINX strategy.
• Significantly enhances OCR abilities by integrating data that requires the model to transcribe text in an image or a document. Also improves abilities in answering questions on charts, figures, and documents with appropriate training data.
• Departs from Idefics1's architecture (gated cross-attentions) and simplifies the integration of visual features into the language backbone. Images are fed to the vision encoder followed by a learned Perceiver pooling and a MLP modality projection. The pooled sequence is then concatenated with the text embeddings to obtain an (interleaved) sequence of image(s) and text(s).
• All these improvements, along with better pre-trained backbones, result in a significant performance jump over Idefics1 for a model that is 10x smaller.

Idefics2 is trained in 2 stages for maximum efficiency. In the first stage, images are fed to the model at SigLIP's native resolution (squares of 384 x 384). In the second stage, images are fed to the model at their native resolution (with a maximum of 980 and a minimum of 378) and native aspect ratio. Since high resolution is necessary for OCR data, PDFA, Rendered-Text, and IDL are added to OBELICS, LAION Coco, and PMD during the second stage.

Following this, instruction fine-tuning is performed on The Cauldron, a collection of 50 manually curated vision-language datasets along with 9 text-only instruction fine-tuning datasets:

• OpenHermes-2.5
• lima
• databricks-dolly-15k
• MetaMathQA
• MathInstruct
• orca-math-word-problems-200k
• math
• atlas-math-sets
• goat

Lora is used to train the parameters initialized from pre-trained backbones, and full fine-tuning is used for newly initialized parameters (modality connector), as this strategy is found to be more stable and computationally efficient.

More details (training procedure, data selection, hyper-parameters, etc.) along with lessons learned from ablations will be available in an upcoming technical report.

🔧 Technical Details

Idefics2 is trained in two stages. In the first stage, images are fed at SigLIP's native resolution (384x384 squares). In the second stage, images are fed at their native resolution (max 980, min 378) and aspect ratio. High-resolution data like PDFA, Rendered-Text, and IDL are added to OBELICS, LAION Coco, and PMD for OCR training.

Instruction fine-tuning is done on The Cauldron and 9 text-only datasets. Lora is used for pre-trained backbone parameters, and full fine-tuning for new parameters.

📄 License

The model is released under the Apache 2.0 license. We release 2 checkpoints:

• idefics2-8b-base: the base model
• idefics2-8b: the base model fine-tuned on a mixture of supervised and instruction datasets (text-only and multimodal datasets)
• idefics2-8b-chatty: idefics2-8b further fine-tuned on long conversation

⚠️ Important Note

Idefics2 will NOT work with Transformers version between 4.41.0 and 4.43.3 included. See the issue https://github.com/huggingface/transformers/issues/32271 and the fix https://github.com/huggingface/transformers/pull/32275.

💡 Usage Tip

As of April 18th, 2024, Idefics2 is part of the 4.40.0 Transformers pypi release. Please upgrade your Transformers version (pip install transformers --upgrade).