Moondream2

🚀 Moondream

Moondream is a compact vision language model engineered for efficient performance across diverse environments. It offers seamless integration of image and text processing capabilities, enabling users to perform various tasks such as captioning, visual querying, object detection, and more.

Website / Demo / GitHub

This repository houses the latest release (2025-04-14) of Moondream, along with historical releases. Given the model's frequent updates, it is advisable to specify a revision as demonstrated below when deploying it in a production application.

🚀 Quick Start

Prerequisites

• Python 3.x
• transformers library
• Pillow library

Installation

You can install the required libraries using pip:

pip install transformers pillow

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
from PIL import Image

model = AutoModelForCausalLM.from_pretrained(
    "vikhyatk/moondream2",
    revision="2025-04-14",
    trust_remote_code=True,
    # Uncomment to run on GPU.
    # device_map={"": "cuda"}
)

# Captioning
print("Short caption:")
print(model.caption(image, length="short")["caption"])

print("\nNormal caption:")
for t in model.caption(image, length="normal", stream=True)["caption"]:
    # Streaming generation example, supported for caption() and detect()
    print(t, end="", flush=True)
print(model.caption(image, length="normal"))

# Visual Querying
print("\nVisual query: 'How many people are in the image?'")
print(model.query(image, "How many people are in the image?")["answer"])

# Object Detection
print("\nObject detection: 'face'")
objects = model.detect(image, "face")["objects"]
print(f"Found {len(objects)} face(s)")

# Pointing
print("\nPointing: 'person'")
points = model.point(image, "person")["points"]
print(f"Found {len(points)} person(s)")

📚 Documentation

Changelog

2025-04-15 (full release notes)

1. Enhanced chart understanding (ChartQA score increased from 74.8 to 77.5, reaching 82.2 with PoT).
2. Implemented temperature and nucleus sampling to mitigate repetitive outputs.
3. Improved OCR for documents and tables (use prompts like “Transcribe the text” or “Transcribe the text in natural reading order”).
4. Extended object detection to support document layout detection (e.g., figures, formulas, text).
5. Boosted UI understanding (ScreenSpot F1@0.5 score rose from 53.3 to 60.3).
6. Strengthened text understanding (DocVQA score increased from 76.5 to 79.3, TextVQA score from 74.6 to 76.3).

2025-03-27 (full release notes)

1. Added support for long-form captioning.
2. Enabled open vocabulary image tagging.
3. Improved counting accuracy (e.g., CountBenchQA score increased from 80 to 86.4).
4. Enhanced text understanding (e.g., OCRBench score increased from 58.3 to 61.2).
5. Optimized object detection, particularly for small objects (e.g., COCO score rose from 30.5 to 51.2).
6. Fixed a token streaming bug affecting multi-byte unicode characters.
7. Supported the gpt-fast style compile() in the HF Transformers implementation.

📄 License

This project is licensed under the Apache-2.0 license.