git-base-captioning-ft-hl-narratives Open-source Image Captioning Model - Generate High-level Narrative Descriptions

Git Base Captioning Ft Hl Narratives

Developed by michelecafagna26

This model is an image caption generation model based on the GIT architecture, fine-tuned on the HL Narratives dataset, specifically designed for generating high-level narrative image descriptions.

Image-to-Text

Transformers

Supports Multiple LanguagesOpen Source License:Apache-2.0 #Narrative image captioning #High-level semantic generation #Social media content creation

Downloads 19

Release Time : 7/24/2023

Model Overview

The model can generate narrative natural language descriptions based on input images, excelling particularly in capturing high-level semantics and storytelling elements within images.

Model Features

Narrative Description Generation

Capable of generating story-driven and context-aware image descriptions, going beyond simple object recognition

High-Quality Fine-Tuning

Fine-tuned on the specialized HL Narratives dataset, enhancing the narrative quality of generated descriptions

Efficient Inference

Supports mixed-precision (fp16) inference, improving inference efficiency

Model Capabilities

Image caption generation

Narrative text generation

Visual content understanding

Use Cases

Social Media

Automatic Image Captioning

Automatically generates story-rich descriptions for social media images

Produces descriptions like 'She is posing for photos at the beach, planning to post them on social media'

Assistive Technology

Visual Assistance

Provides richer image descriptions for visually impaired individuals

🚀 GIT-base fine-tuned for Narrative Image Captioning

This is a fine-tuned version of the GIT base model. It is trained on the HL Narratives dataset for generating high-level narrative descriptions of images.

✨ Features

High-Level Narrative: Capable of generating high-level narrative descriptions for images.
Fine-Tuned Model: Based on the GIT base model, fine-tuned on HL Narratives dataset.

📦 Installation

The provided README doesn't contain specific installation steps, so this section is skipped.

💻 Usage Examples

Basic Usage

import requests
from PIL import Image
from transformers import AutoProcessor, AutoModelForCausalLM

processor = AutoProcessor.from_pretrained("git-base-captioning-ft-hl-narratives")
model = AutoModelForCausalLM.from_pretrained("git-base-captioning-ft-hl-narratives").to("cuda")

img_url = 'https://datasets-server.huggingface.co/assets/michelecafagna26/hl/--/default/train/0/image/image.jpg' 
raw_image = Image.open(requests.get(img_url, stream=True).raw).convert('RGB')

inputs = processor(raw_image, return_tensors="pt").to("cuda")
pixel_values = inputs.pixel_values

generated_ids = model.generate(pixel_values=pixel_values, max_length=50,
            do_sample=True,
            top_k=120,
            top_p=0.9,
            early_stopping=True,
            num_return_sequences=1)

processor.batch_decode(generated_ids, skip_special_tokens=True)

>>> "she is posing for a photo on the beach, she wants to post on her social media."

📚 Documentation

Model fine-tuning 🏋️‍

Trained for 3 epochs.
Learning rate (lr): 5e−5.
Adam optimizer.
Half-precision (fp16).

Test set metrics 🧾

Cider	SacreBLEU	Rouge-L
75.78	11.11	27.61

🔧 Technical Details

The provided README doesn't contain specific technical details beyond what's already in the documentation section, so this section is skipped.

📄 License

The model is licensed under the Apache-2.0 license.

BibTex and citation info

@inproceedings{cafagna2023hl,
  title={{HL} {D}ataset: {V}isually-grounded {D}escription of {S}cenes, {A}ctions and
{R}ationales},
  author={Cafagna, Michele and van Deemter, Kees and Gatt, Albert},
  booktitle={Proceedings of the 16th International Natural Language Generation Conference (INLG'23)},
address = {Prague, Czech Republic},
  year={2023}
}

Featured Recommended AI Models

Empowering the Future, Your AI Solution Knowledge Base

English 简体中文繁體中文にほんご