Git Base Finetune

🚀 GIT (GenerativeImage2Text), base-sized

GIT (GenerativeImage2Text) is a base-sized model designed for vision and language tasks. It can convert images into text, offering solutions for image captioning, visual question - answering, and more.

🚀 Quick Start

This model can be used for image captioning. You can check the model hub to find fine - tuned versions for specific tasks. For code examples, refer to the documentation.

✨ Features

• Versatile Applications: Suitable for various vision - language tasks such as image and video captioning, visual question answering on images and videos, and even image classification.
• Transformer - based: A Transformer decoder that takes both CLIP image tokens and text tokens into account.

📦 Installation

No specific installation steps are provided in the original document, so this section is skipped.

💻 Usage Examples

For code examples, please refer to the documentation.

📚 Documentation

Model description

GIT is a Transformer decoder conditioned on both CLIP image tokens and text tokens. The model is trained using "teacher forcing" on a large number of (image, text) pairs.

The main goal of the model is to predict the next text token, given the image tokens and previous text tokens. When predicting the next text token, the model has full access to the image patch tokens (using a bidirectional attention mask), but only has access to the previous text tokens (using a causal attention mask for the text tokens).

This enables the model to be used for tasks like:

• Image and video captioning
• Visual question answering (VQA) on images and videos
• Image classification (by conditioning the model on the image and asking it to generate a class in text)

Intended uses & limitations

You can use the raw model for image captioning. Check the model hub for fine - tuned versions for tasks that interest you.

Training data

From the paper:

We collect 0.8B image - text pairs for pre - training, which include COCO (Lin et al., 2014), Conceptual Captions (CC3M) (Sharma et al., 2018), SBU (Ordonez et al., 2011), Visual Genome (VG) (Krishna et al., 2016), Conceptual Captions (CC12M) (Changpinyo et al., 2021), ALT200M (Hu et al., 2021a), and an extra 0.6B data following a similar collection procedure in Hu et al. (2021a).

However, this is for the model referred to as "GIT" in the paper, which is not open - sourced. This checkpoint is "GIT - base", a smaller variant of GIT trained on 10 million image - text pairs. See table 11 in the paper for more details.

Preprocessing

Refer to the original repo for details on preprocessing during training. During validation, resize the shorter edge of each image and then perform center cropping to a fixed - size resolution. Next, normalize the frames across the RGB channels using the ImageNet mean and standard deviation.

Evaluation results

For evaluation results, refer to the paper.

🔧 Technical Details

GIT is a Transformer - based decoder. It uses "teacher forcing" for training on (image, text) pairs. The attention mechanism for image patch tokens and text tokens is different, with a bidirectional attention mask for image patch tokens and a causal attention mask for text tokens.

📄 License

This model is released under the MIT license.

⚠️ Important Note

The team releasing GIT did not write a model card for this model, so this model card has been written by the Hugging Face team.