Rgb Language Cap

🚀 Spatial Aware Vision-Language (VL) Model

We are creating a spatial aware vision - language (VL) model that can generate captions for images with spatial relationship information.

🚀 Quick Start

This is a sequence - to - sequence model for image - captioning. It's been trained on the COCO dataset images, incorporating extra information about the spatial relationship between the entities in the image. The architecture consists of a ViT encoder and a GPT2 decoder.

✨ Features

• The model is trained to generate as many words as possible, with a maximum of 200 tokens (roughly 5 sentences, and the 6th sentence is usually cropped).
• The output always follows the form: "Object1" is to the "Left/Right etc." of the "Object2".

📦 Installation

💻 Usage Examples

Basic Usage

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
from transformers import pipeline
image_captioner = pipeline("image-to-text", model="sadassa17/rgb-language_cap", max_new_tokens=200, device=device)
filename = 'path/to/file'
generated_captions = image_captioner(filename)
print(generated_captions)

Advanced Usage

If you want to produce a specific number of captions up to 5:

import os
def print_up_to_n_sentences(captions, n):
    for caption in captions:
        generated_text = caption.get('generated_text', '')
        sentences = generated_text.split('.')
        result = '.'.join(sentences[:n])
        #print(result)
    return result
filename = 'path/to/file'

generated_captions = image_captioner(filename)
captions = print_up_to_n_sentences(generated_captions, 5)
print(captions)

📄 License

This project is licensed under the MIT license.