Zoedepth-NYU-Kitti Open-Source Depth Estimation Model - Precise Depth Value Estimation in Real Metrics

Zoedepth Nyu Kitti

Developed by Intel

ZoeDepth is a depth estimation model fine-tuned on NYU and KITTI datasets, capable of estimating depth values in actual metric units.

3D Vision

Transformers

Open Source License:MIT #Monocular depth estimation #Zero-shot transfer #Metric depth prediction

Downloads 20.32k

Release Time : 4/30/2024

Model Overview

This model extends the DPT framework for metric (absolute) depth estimation, excelling in zero-shot monocular depth estimation tasks.

Model Features

Metric depth estimation

Capable of estimating depth values in actual metric units (e.g., meters), not just relative depth.

Zero-shot transfer capability

Combines relative and metric depth estimation methods, demonstrating strong zero-shot transfer performance.

State-of-the-art performance

Achieves state-of-the-art performance levels in depth estimation tasks.

Model Capabilities

Monocular depth estimation

Zero-shot transfer learning

Metric depth prediction

Use Cases

Computer vision

3D scene reconstruction

Estimates scene depth information from a single image for 3D reconstruction.

Generates precise depth maps.

Autonomous driving perception

Used for environmental perception and obstacle detection in autonomous driving systems.

Provides accurate depth information to aid navigation decisions.

Augmented reality

Supplies scene depth information for AR applications.

Enables more realistic virtual object overlays.

🚀 ZoeDepth (fine-tuned on NYU and KITTI)

ZoeDepth model fine-tuned on the NYU and KITTI datasets, achieving state-of-the-art results in metric depth estimation.

🚀 Quick Start

ZoeDepth extends the DPT framework for metric (also called absolute) depth estimation. It was introduced in the paper ZoeDepth: Zero-shot Transfer by Combining Relative and Metric Depth by Shariq et al. and first released in this repository.

Disclaimer: The team releasing ZoeDepth did not write a model card for this model so this model card has been written by the Hugging Face team.

✨ Features

Adapts DPT for metric depth estimation.
Capable of estimating depth in actual metric values.
Achieves state-of-the-art results in depth estimation tasks.

📚 Documentation

Model description

ZoeDepth adapts DPT, a model for relative depth estimation, for so-called metric (also called absolute) depth estimation. This means that the model is able to estimate depth in actual metric values.

ZoeDepth architecture

ZoeDepth architecture. Taken from the original paper.

Intended uses & limitations

You can use the raw model for tasks like zero-shot monocular depth estimation. See the model hub to look for other versions on a task that interests you.

💻 Usage Examples

Basic Usage

The easiest is to leverage the pipeline API which abstracts away the complexity for the user:

from transformers import pipeline
from PIL import Image
import requests

# load pipe
depth_estimator = pipeline(task="depth-estimation", model="Intel/zoedepth-nyu-kitti")

# load image
url = 'http://images.cocodataset.org/val2017/000000039769.jpg'
image = Image.open(requests.get(url, stream=True).raw)

# inference
outputs = depth_estimator(image)
depth = outputs.depth

For more code examples, we refer to the documentation.

BibTeX entry and citation info

@misc{bhat2023zoedepth,
      title={ZoeDepth: Zero-shot Transfer by Combining Relative and Metric Depth}, 
      author={Shariq Farooq Bhat and Reiner Birkl and Diana Wofk and Peter Wonka and Matthias Müller},
      year={2023},
      eprint={2302.12288},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}

📄 License

This project is licensed under the MIT license.

Featured Recommended AI Models

Empowering the Future, Your AI Solution Knowledge Base

English 简体中文繁體中文にほんご