ZoeDepth-NYU Open-Source Monocular Depth Estimation Model - Achieving Accurate Metric Depth Estimation via Zero-Shot Transfer

Zoedepth Nyu

Developed by Intel

ZoeDepth is a model for monocular depth estimation, specifically fine-tuned on the NYU dataset, capable of zero-shot transfer and metric depth estimation.

3D Vision

Transformers

Open Source License:MIT #Monocular depth estimation #Zero-shot transfer #Metric depth

Downloads 1,279

Release Time : 4/30/2024

Model Overview

ZoeDepth extends the DPT framework for metric (absolute) depth estimation, achieving state-of-the-art results. The model can perform monocular depth estimation in a zero-shot manner and outputs depth information with actual metric values.

Model Features

Zero-shot transfer

Capable of performing depth estimation without task-specific training data.

Metric depth estimation

Can output actual metric depth values, not just relative depth.

Based on the DPT framework

Extends the DPT framework, combining the advantages of relative and metric depth estimation.

Model Capabilities

Monocular depth estimation

Zero-shot transfer

Metric depth output

Use Cases

Computer vision

Indoor scene depth estimation

The model fine-tuned on the NYU dataset is particularly suitable for depth estimation in indoor scenes.

Can accurately estimate depth information in indoor environments.

3D scene reconstruction

Can be used to reconstruct 3D scenes from a single image.

Provides precise depth information to support 3D reconstruction.

🚀 ZoeDepth (fine-tuned on NYU)

A fine-tuned ZoeDepth model on the NYU dataset for depth estimation, achieving state-of-the-art results.

🚀 Quick Start

ZoeDepth is a model fine - tuned on the NYU dataset. It was introduced in the paper ZoeDepth: Zero-shot Transfer by Combining Relative and Metric Depth by Shariq et al. and first released in this repository. It extends the DPT framework for metric (also called absolute) depth estimation.

Disclaimer: The team releasing ZoeDepth did not write a model card for this model, so this model card has been written by the Hugging Face team.

✨ Features

Adapts DPT, a relative depth estimation model, for metric (absolute) depth estimation.
Capable of estimating depth in actual metric values.

📚 Documentation

Model description

ZoeDepth adapts DPT, a model for relative depth estimation, for so - called metric (also called absolute) depth estimation. This means that the model is able to estimate depth in actual metric values.

ZoeDepth architecture

ZoeDepth architecture. Taken from the original paper.

Intended uses & limitations

You can use the raw model for tasks like zero - shot monocular depth estimation. See the model hub to look for other versions on a task that interests you.

💻 Usage Examples

Basic Usage

The easiest is to leverage the pipeline API which abstracts away the complexity for the user:

from transformers import pipeline
from PIL import Image
import requests

# load pipe
depth_estimator = pipeline(task="depth-estimation", model="Intel/zoedepth-nyu")

# load image
url = 'http://images.cocodataset.org/val2017/000000039769.jpg'
image = Image.open(requests.get(url, stream=True).raw)

# inference
outputs = depth_estimator(image)
depth = outputs.depth

For more code examples, we refer to the documentation.

BibTeX entry and citation info

@misc{bhat2023zoedepth,
      title={ZoeDepth: Zero-shot Transfer by Combining Relative and Metric Depth}, 
      author={Shariq Farooq Bhat and Reiner Birkl and Diana Wofk and Peter Wonka and Matthias Müller},
      year={2023},
      eprint={2302.12288},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}

📄 License

This model is released under the MIT license.

Featured Recommended AI Models

Empowering the Future, Your AI Solution Knowledge Base

English 简体中文繁體中文にほんご