Moondream Caption

🚀 Moondream-Caption: Custom Small Vision Model based on Moondream2

Moondream-Caption is a custom small vision model that enhances image description capabilities by fine - tuning on a specific dataset, based on moondream2 by vikhyatk.

🚀 Quick Start

You can use Moondream-Caption for image captioning tasks by leveraging the Hugging Face Transformers library. Here's a quick example of how to generate captions for an image:

from transformers import AutoTokenizer, AutoModelForCausalLM
from PIL import Image

moondream = AutoModelForCausalLM.from_pretrained(
   "wraps/moondream-caption", trust_remote_code=True
)
tokenizer = AutoTokenizer.from_pretrained("wraps/moondream-caption")

image = Image.open("path/to/your/image.jpg")
enc_image = moondream.encode_image(image)
caption = model.answer_question(enc_image, "Write a long caption for this image")

print(caption)

✨ Features

• Based on the moondream2 architecture
• Fine-tuned for image caption generation
• Trained on a high-quality custom dataset

📦 Installation

No specific installation steps are provided in the original document.

💻 Usage Examples

Basic Usage

from transformers import AutoTokenizer, AutoModelForCausalLM
from PIL import Image

moondream = AutoModelForCausalLM.from_pretrained(
   "wraps/moondream-caption", trust_remote_code=True
)
tokenizer = AutoTokenizer.from_pretrained("wraps/moondream-caption")

image = Image.open("path/to/your/image.jpg")
enc_image = moondream.encode_image(image)
caption = model.answer_question(enc_image, "Write a long caption for this image")

print(caption)

Example

Output Caption: A close-up portrait of a green alien with a large oval head, enormous black almond-shaped eyes, small nostrils, and a tiny mouth. The alien has a long, thin neck and is wearing a black t-shirt with white text that reads 'humans scare me'. The background shows a pale blue sky with soft, wispy clouds.

📚 Documentation

Dataset

The dataset used for training Moondream-Caption is specifically designed for image captioning tasks. It has the following characteristics:

• Images generated with flux1_dev
• Highly accurate and verified descriptive captions
• Wide variety of visual content

Limitations

While Moondream-Caption is designed to generate accurate and relevant image captions, it may not perform optimally on images that significantly differ from the training dataset. Additionally, the model may struggle with complex or abstract images that deviate from the dataset's content. Please open an issue on the model's repository if you encounter any limitations or issues.

📄 License

The project is licensed under the Apache-2.0 license.