đ Describe Anything: Detailed Localized Image and Video Captioning
NVIDIA, UC Berkeley, UCSF present a model that enables detailed localized descriptions of images and videos, facilitating research and non - commercial development.
Authors: Long Lian, Yifan Ding, Yunhao Ge, Sifei Liu, Hanzi Mao, Boyi Li, Marco Pavone, Ming - Yu Liu, Trevor Darrell, Adam Yala, Yin Cui
[Paper] | [Code] | [Project Page] | [Video] | [HuggingFace Demo] | [Model/Benchmark/Datasets] | [Citation]
đ Quick Start
The Describe Anything Model 3B Video (DAM - 3B - Video) is designed to generate detailed localized descriptions of images and videos. It's mainly for research and non - commercial use. You can access the model through various links provided above, such as the HuggingFace Demo for a quick test.
⨠Features
- Localized Description: DAM - 3B - Video can generate detailed descriptions for user - specified regions in images/videos using points/boxes/scribbles/masks.
- Novel Architecture: It integrates full - image/video context with fine - grained local details via a novel focal prompt and a localized vision backbone enhanced with gated cross - attention.
đĻ Installation
No specific installation steps are provided in the original README.
đģ Usage Examples
No code examples are provided in the original README.
đ Documentation
Model Card for DAM - 3B
Description
Describe Anything Model 3B Video (DAM - 3B - Video) takes inputs of user - specified regions in the form of points/boxes/scribbles/masks within images/videos, and generates detailed localized descriptions of images/videos. DAM integrates full - image/video context with fine - grained local details using a novel focal prompt and a localized vision backbone enhanced with gated cross - attention. The model is for research and development only. This model is ready for non - commercial use.
License
NVIDIA Noncommercial License
Intended Usage
This model is intended to demonstrate and facilitate the understanding and usage of the describe anything models. It should primarily be used for research and non - commercial purposes.
Model Architecture
| Property |
Details |
| Architecture Type |
Transformer |
| Network Architecture |
ViT and Llama |
This model was developed based on VILA - 1.5. This model has 3B of model parameters.
Input
| Property |
Details |
| Input Type(s) |
Image, Video, Text, Binary Mask |
| Input Format(s) |
RGB Image, RGB Video, Binary Mask |
| Input Parameters |
2D Image, 2D Video, 2D Binary Mask |
| Other Properties Related to Input |
3 channels for RGB image, 3 channels for RGB video, 1 channel for binary mask. Resolution is 384x384. |
Output
| Property |
Details |
| Output Type(s) |
Text |
| Output Format |
String |
| Output Parameters |
1D Text |
| Other Properties Related to Output |
Detailed descriptions for the visual region. |
Supported Hardware and OS
Supported Hardware Microarchitecture Compatibility:
- NVIDIA Ampere
- NVIDIA Hopper
- NVIDIA Lovelace
Preferred/Supported Operating System(s):
Training Dataset
Describe Anything Training Datasets
Evaluation Dataset
We evaluate our models on our detailed localized captioning benchmark: DLC - Bench
Inference
PyTorch
Ethical Considerations
NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. When downloaded or used in accordance with our terms of service, developers should work with their internal model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse.
Please report security vulnerabilities or NVIDIA AI Concerns here.
đ§ Technical Details
The model uses a novel focal prompt and a localized vision backbone enhanced with gated cross - attention to integrate full - image/video context with fine - grained local details. It is developed based on VILA - 1.5 and has 3B of model parameters.
đ License
NVIDIA Noncommercial License
Citation
If you use our work or our implementation in this repo, or find them helpful, please consider giving a citation.
@article{lian2025describe,
title={Describe Anything: Detailed Localized Image and Video Captioning},
author={Long Lian and Yifan Ding and Yunhao Ge and Sifei Liu and Hanzi Mao and Boyi Li and Marco Pavone and Ming - Yu Liu and Trevor Darrell and Adam Yala and Yin Cui},
journal={arXiv preprint arXiv:2504.16072},
year={2025}
}
%20--%3e%3cdefs%3e%3cstyle%3e%20.st0%20{%20fill:%20%23061b40;%20}%20.st1%20{%20fill:%20%23306af1;%20}%20.st2%20{%20fill:%20%235ce5cf;%20}%20%3c/style%3e%3c/defs%3e%3cg%3e%3cpath%20class='st0'%20d='M55,10.5h9v3h-9v9h12V7.5h-12v3ZM64,19.5h-6v-3h6v3Z'/%3e%3cpolygon%20class='st0'%20points='69%2016.5%2078%2016.5%2078%2019.5%2069%2019.5%2069%2022.5%2081%2022.5%2081%2013.5%2072%2013.5%2072%2010.5%2081%2010.5%2081%207.5%2069%207.5%2069%2016.5'/%3e%3cpolygon%20class='st0'%20points='95%2010.5%2095%207.5%2083%207.5%2083%2022.5%2095%2022.5%2095%2019.5%2086%2019.5%2086%2016.5%2095%2016.5%2095%2013.5%2086%2013.5%2086%2010.5%2095%2010.5'/%3e%3cpath%20class='st0'%20d='M40,1.5v21h11.6l1.4-1.4v-7.6h0c0,0-1.4-1.5-1.4-1.5l1.4-1.4V2.9l-1.4-1.4h-11.6ZM50,19.5h-7v-6h7v6ZM50,10.5h-7v-6h7v6Z'/%3e%3c/g%3e%3cpath%20class='st1'%20d='M23.1,24L14.7,4.8l-4.9,11.2h3.8l-1.8,4H3.3L12.1,0H2C.9,0,0,.9,0,2v20c0,1.1.9,2,2,2h21.1Z'/%3e%3cpath%20class='st2'%20d='M34,0h-16.8l10.6,24h6.2c1.1,0,2-.9,2-2V2C36,.9,35.1,0,34,0ZM32.5,20h-4V4h4v16Z'/%3e%3c/svg%3e)
Transformers