Blip2 Flan T5 Xl Sharded

🚀 Sharded BLIP-2 Model Card - flan-t5-xl

This is a sharded version of the blip2-flan-t5-xl, which utilizes Flan T5-xl for image-to-text tasks like image captioning and visual question answering.

🚀 Quick Start

This sharded model can be easily loaded on low-RAM Colab runtimes. For more details about the model description, intended uses, and limitations, as well as instructions for using the model on CPU and GPU in different precisions, refer to the original model card.

✨ Features

• The model repo is sharded, enabling easy loading on low-RAM Colab runtimes.

📦 Installation

Requires the current main of transformers (at time of writing):

pip install accelerate git+https://github.com/huggingface/transformers.git -U -q

💻 Usage Examples

Basic Usage

Here is how you can use it on CPU. Note that this is for CPU; check out the original model card/blog for fp16 and int8 usage.

import requests
from PIL import Image
from transformers import BlipProcessor, Blip2ForConditionalGeneration

model_name = "ethzanalytics/blip2-flan-t5-xl-sharded"
processor = BlipProcessor.from_pretrained(model_name)
model = Blip2ForConditionalGeneration.from_pretrained(model_name)

img_url = 'https://storage.googleapis.com/sfr-vision-language-research/BLIP/demo.jpg' 
raw_image = Image.open(requests.get(img_url, stream=True).raw).convert('RGB')

question = "how many dogs are in the picture?"
inputs = processor(raw_image, question, return_tensors="pt")

out = model.generate(**inputs)
print(processor.decode(out[0], skip_special_tokens=True))

📄 License

This project is licensed under the MIT license.