Git Large R Textcaps

🚀 GIT (GenerativeImage2Text), large-sized, fine-tuned on TextCaps, R*

The GIT model is designed for image - to - text tasks, offering high - quality captioning and related capabilities.

🚀 Quick Start

The GIT (GenerativeImage2Text) model, in its large - sized version, is fine - tuned on TextCaps. It was introduced in the paper GIT: A Generative Image - to - text Transformer for Vision and Language by Wang et al. and first released in this repository.

Disclaimer: The team releasing GIT did not write a model card for this model so this model card has been written by the Hugging Face team.

✨ Features

GIT is a Transformer decoder conditioned on both CLIP image tokens and text tokens. The model is trained using "teacher forcing" on a large number of (image, text) pairs. Its goal is to predict the next text token given the image tokens and previous text tokens. The model has full access to the image patch tokens (using a bidirectional attention mask) and only access to the previous text tokens (using a causal attention mask) when predicting the next text token.

This enables the model to be used for various tasks:

• Image and video captioning
• Visual question answering (VQA) on images and videos
• Even image classification (by conditioning the model on the image and asking it to generate a class in text)

📚 Documentation

Intended uses & limitations

You can use the raw model for image captioning. Check the model hub to find fine - tuned versions for tasks that interest you.

How to use

For code examples, refer to the documentation.

📦 Installation

No installation steps are provided in the original document, so this section is skipped.

💻 Usage Examples

No code examples are provided in the original document, so this section is skipped.

🔧 Technical Details

Training data

From the paper:

We collect 0.8B image - text pairs for pre - training, which include COCO (Lin et al., 2014), Conceptual Captions (CC3M) (Sharma et al., 2018), SBU (Ordonez et al., 2011), Visual Genome (VG) (Krishna et al., 2016), Conceptual Captions (CC12M) (Changpinyo et al., 2021), ALT200M (Hu et al., 2021a), and an extra 0.6B data following a similar collection procedure in Hu et al. (2021a).

However, this is for the model referred to as "GIT" in the paper, which is not open - sourced. This checkpoint is "GIT - large", a smaller variant of GIT trained on 20 million image - text pairs. Then, the model was fine - tuned on TextCaps. See table 11 in the paper for more details.

Preprocessing

Refer to the original repo for details on preprocessing during training. During validation, the shorter edge of each image is resized, followed by center cropping to a fixed - size resolution. Next, frames are normalized across the RGB channels with the ImageNet mean and standard deviation.

📄 License

This model is released under the MIT license.