Rgb Language Cap

🚀 Spatial Aware Vision-Language (VL) Model

We are creating a spatial aware vision - language (VL) model that can generate captions for images with spatial relationship information.

🚀 Quick Start

This is a trained model on the COCO dataset images, which includes extra information about the spatial relationship between the entities in the image. It is a sequence - to - sequence model for image - captioning, with a ViT encoder and a GPT2 decoder architecture.

Requirements

Download and Run

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
from transformers import pipeline
image_captioner = pipeline("image-to-text", model="voxreality/rgb-language_cap", max_new_tokens=200, device=device)
filename = 'path/to/file'
generated_captions = image_captioner(filename)
print(generated_captions)

The model is trained to generate as many words as possible, with a maximum of 200 tokens, which is approximately 5 sentences. Usually, the 6th sentence is cropped. The output always follows the form: "Object1" is to the "Left/Right etc." of the "Object2".

💻 Usage Examples

Basic Usage

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
from transformers import pipeline
image_captioner = pipeline("image-to-text", model="voxreality/rgb-language_cap", max_new_tokens=200, device=device)
filename = 'path/to/file'
generated_captions = image_captioner(filename)
print(generated_captions)

Advanced Usage

If you want to produce a specific number of captions up to 5:

import os
def print_up_to_n_sentences(captions, n):
    for caption in captions:
        generated_text = caption.get('generated_text', '')
        sentences = generated_text.split('.')
        result = '.'.join(sentences[:n])
        #print(result)
    return result
filename = 'path/to/file'

generated_captions = image_captioner(filename)
captions = print_up_to_n_sentences(generated_captions, 5)
print(captions)

📄 License

This project is licensed under the Apache - 2.0 license.