blip-image-captioning-base-mocha Open-Source Image Captioning Model - Alleviate Hallucination Issues and Accurately Describe Images

Blip Image Captioning Base Mocha

Developed by moranyanuka

Official checkpoint of BLIP base model fine-tuned on MS-COCO dataset using MOCHA reinforcement learning framework to mitigate open-vocabulary description hallucination

Image-to-Text

Transformers

Open Source License:MIT #Anti-hallucination image captioning #Reinforcement learning fine-tuning #Open-vocabulary generation

Downloads 88

Release Time : 12/19/2023

Model Overview

This model is an image-to-text generation model based on BLIP architecture, specifically designed for generating image captions. Fine-tuned with the MOCHA reinforcement learning framework, it effectively reduces hallucination issues in descriptions.

Model Features

MOCHA reinforcement learning fine-tuning

Fine-tuned with MOCHA framework to effectively mitigate hallucination issues in open-vocabulary descriptions

Dual-mode generation

Supports both conditional and unconditional image caption generation

Multi-precision support

Can run on CPU/GPU with support for both full precision and half precision (float16) modes

Model Capabilities

Image caption generation

Conditional text generation

Unconditional text generation

Multilingual image understanding

Use Cases

Content generation

Automatic image tagging

Automatically generates descriptive text for images in social media or content management systems

Produces accurate, hallucination-free image captions

Assisting visually impaired users

Provides textual descriptions of image content for visually impaired users

Enhances accessibility and aids in understanding visual content

Computer vision research

Vision-language model research

Serves as baseline or comparative model for vision-language tasks

Provides benchmark performance optimized by MOCHA

🚀 Mocha Checkpoint for BLIP-Base Model

This is the official checkpoint of the BLIP-Base model, finetuned on MS-COCO using the MOCHa RL framework. It was introduced in Mitigating Open-Vocabulary Caption Hallucinations.

Project Page

🚀 Quick Start

You can use this model for conditional and un-conditional image captioning.

💻 Usage Examples

Basic Usage

Running the model on CPU

Click to expand

import requests
from PIL import Image
from transformers import BlipProcessor, BlipForConditionalGeneration

processor = BlipProcessor.from_pretrained("moranyanuka/blip-image-captioning-base-mocha")
model = BlipForConditionalGeneration.from_pretrained("moranyanuka/blip-image-captioning-base-mocha")

img_url = 'https://storage.googleapis.com/sfr-vision-language-research/BLIP/demo.jpg' 
raw_image = Image.open(requests.get(img_url, stream=True).raw).convert('RGB')

# conditional image captioning
text = "a photography of"
inputs = processor(raw_image, text, return_tensors="pt")

out = model.generate(**inputs)
print(processor.decode(out[0], skip_special_tokens=True))

# unconditional image captioning
inputs = processor(raw_image, return_tensors="pt")

out = model.generate(**inputs)
print(processor.decode(out[0], skip_special_tokens=True))

Advanced Usage

Running the model on GPU

In full precision

Click to expand

import requests
from PIL import Image
from transformers import BlipProcessor, BlipForConditionalGeneration

processor = BlipProcessor.from_pretrained("moranyanuka/blip-image-captioning-base-mocha")
model = BlipForConditionalGeneration.from_pretrained("moranyanuka/blip-image-captioning-base-mocha").to("cuda")

img_url = 'https://storage.googleapis.com/sfr-vision-language-research/BLIP/demo.jpg' 
raw_image = Image.open(requests.get(img_url, stream=True).raw).convert('RGB')

# conditional image captioning
text = "a photography of"
inputs = processor(raw_image, text, return_tensors="pt").to("cuda")

out = model.generate(**inputs)
print(processor.decode(out[0], skip_special_tokens=True))

# unconditional image captioning
inputs = processor(raw_image, return_tensors="pt").to("cuda")

out = model.generate(**inputs)
print(processor.decode(out[0], skip_special_tokens=True))

In half precision (`float16`)

Click to expand

import torch
import requests
from PIL import Image
from transformers import BlipProcessor, BlipForConditionalGeneration

processor = BlipProcessor.from_pretrained("moranyanuka/blip-image-captioning-base-mocha")
model = BlipForConditionalGeneration.from_pretrained("moranyanuka/blip-image-captioning-base-mocha", torch_dtype=torch.float16).to("cuda")

img_url = 'https://storage.googleapis.com/sfr-vision-language-research/BLIP/demo.jpg' 
raw_image = Image.open(requests.get(img_url, stream=True).raw).convert('RGB')

# conditional image captioning
text = "a photography of"
inputs = processor(raw_image, text, return_tensors="pt").to("cuda", torch.float16)

out = model.generate(**inputs)
print(processor.decode(out[0], skip_special_tokens=True))
# >>> a photography of a woman and her dog on the beach

# unconditional image captioning
inputs = processor(raw_image, return_tensors="pt").to("cuda", torch.float16)

out = model.generate(**inputs)
print(processor.decode(out[0], skip_special_tokens=True))
# >>> a woman sitting on the beach with a dog

📄 License

This project is licensed under the MIT license.

📚 Documentation

BibTeX Citation

@misc{benkish2024mitigating,
      title={Mitigating Open-Vocabulary Caption Hallucinations}, 
      author={Assaf Ben-Kish and Moran Yanuka and Morris Alper and Raja Giryes and Hadar Averbuch-Elor},
      year={2024},
      eprint={2312.03631},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}

Featured Recommended AI Models

Empowering the Future, Your AI Solution Knowledge Base

English 简体中文繁體中文にほんご

Blip Image Captioning Base Mocha

Model Overview

Model Features

Model Capabilities

Use Cases

🚀 Mocha Checkpoint for BLIP-Base Model

🚀 Quick Start

💻 Usage Examples

Basic Usage

Running the model on CPU

Advanced Usage

Running the model on GPU

In full precision

In half precision (float16)

📄 License

📚 Documentation

BibTeX Citation

In half precision (`float16`)