Dpt Hybrid Midas

🚀 DPT-Hybrid (MiDaS 3.0)

A Dense Prediction Transformer (DPT) model trained on 1.4 million images for monocular depth estimation.

🚀 Quick Start

You can use the raw model for zero - shot monocular depth estimation. See the model hub to look for fine - tuned versions on a task that interests you.

✨ Features

• Trained on 1.4 million images for monocular depth estimation.
• Uses the Vision Transformer (ViT) as backbone and adds a neck + head for monocular depth estimation.
• The "hybrid" version uses [ViT - hybrid](https://huggingface.co/google/vit - hybrid - base - bit - 384) as a backbone and takes some activations from the backbone.

📦 Installation

No specific installation steps are provided in the original document.

💻 Usage Examples

Basic Usage

Here is how to use this model for zero - shot depth estimation on an image:

from PIL import Image
import numpy as np
import requests
import torch

from transformers import DPTImageProcessor, DPTForDepthEstimation

image_processor = DPTImageProcessor.from_pretrained("Intel/dpt - hybrid - midas")
model = DPTForDepthEstimation.from_pretrained("Intel/dpt - hybrid - midas", low_cpu_mem_usage=True)

url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)

# prepare image for the model
inputs = image_processor(images=image, return_tensors="pt")

with torch.no_grad():
    outputs = model(**inputs)
    predicted_depth = outputs.predicted_depth

# interpolate to original size
prediction = torch.nn.functional.interpolate(
    predicted_depth.unsqueeze(1),
    size=image.size[::-1],
    mode="bicubic",
    align_corners=False,
)

# visualize the prediction
output = prediction.squeeze().cpu().numpy()
formatted = (output * 255 / np.max(output)).astype("uint8")
depth = Image.fromarray(formatted)
depth.show()

For more code examples, we refer to the documentation.

📚 Documentation

Model Details

Factors

Metrics

Training and Evaluation Data

Quantitative Analyses

Table 1. Comparison to the state of the art on monocular depth estimation. We evaluate zero - shot cross - dataset transfer according to the protocol defined in [30]. Relative performance is computed with respect to the original MiDaS model [30]. Lower is better for all metrics. (Ranftl et al., 2021)

Ethical Considerations

Caveats and Recommendations

Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. There are no additional caveats or recommendations for this model.

🔧 Technical Details

The Dense Prediction Transformer (DPT) uses the Vision Transformer (ViT) as backbone and adds a neck + head on top for monocular depth estimation. The "hybrid" version diverges from DPT by using [ViT - hybrid](https://huggingface.co/google/vit - hybrid - base - bit - 384) as a backbone and taking some activations from the backbone.

📄 License

This project is licensed under the Apache 2.0 license.

BibTeX entry and citation info

@article{DBLP:journals/corr/abs-2103-13413,
  author    = {Ren{\'{e}} Ranftl and
               Alexey Bochkovskiy and
               Vladlen Koltun},
  title     = {Vision Transformers for Dense Prediction},
  journal   = {CoRR},
  volume    = {abs/2103.13413},
  year      = {2021},
  url       = {https://arxiv.org/abs/2103.13413},
  eprinttype = {arXiv},
  eprint    = {2103.13413},
  timestamp = {Wed, 07 Apr 2021 15:31:46 +0200},
  biburl    = {https://dblp.org/rec/journals/corr/abs-2103-13413.bib},
  bibsource = {dblp computer science bibliography, https://dblp.org}
}