Depthpro

🚀 Depth Pro: Sharp Monocular Metric Depth in Less Than a Second

Depth Pro is a foundation model for zero - shot metric monocular depth estimation. It can synthesize high - resolution depth maps with sharpness and high - frequency details, and make fast and metric predictions without relying on camera metadata.

🚀 Quick Start

To start using Depth Pro, you need to set up your environment. Follow the steps in the code repository. After that, you can download the checkpoint from the Files and versions tab above, or use the huggingface - hub CLI:

pip install huggingface-hub
huggingface-cli download --local-dir checkpoints apple/DepthPro

✨ Features

• High - Quality Depth Maps: Depth Pro synthesizes high - resolution depth maps with unparalleled sharpness and high - frequency details.
• Metric Predictions: The predictions are metric, with absolute scale, without relying on metadata such as camera intrinsics.
• Fast Inference: It can produce a 2.25 - megapixel depth map in 0.3 seconds on a standard GPU.
• Technical Innovations: Enabled by an efficient multi - scale vision transformer, a special training protocol, dedicated evaluation metrics, and state - of - the - art focal length estimation.

💻 Usage Examples

Basic Usage

Running from commandline

The code repo provides a helper script to run the model on a single image:

# Run prediction on a single image:
depth-pro-run -i ./data/example.jpg
# Run `depth-pro-run -h` for available options.

Running from Python

from PIL import Image
import depth_pro

# Load model and preprocessing transform
model, transform = depth_pro.create_model_and_transforms()
model.eval()

# Load and preprocess an image.
image, _, f_px = depth_pro.load_rgb(image_path)
image = transform(image)

# Run inference.
prediction = model.infer(image, f_px=f_px)
depth = prediction["depth"]  # Depth in [m].
focallength_px = prediction["focallength_px"]  # Focal length in pixels.

Advanced Usage

Evaluation (boundary metrics)

Boundary metrics are implemented in eval/boundary_metrics.py and can be used as follows:

# for a depth-based dataset
boundary_f1 = SI_boundary_F1(predicted_depth, target_depth)

# for a mask-based dataset (image matting / segmentation) 
boundary_recall = SI_boundary_Recall(predicted_depth, target_mask)

📚 Documentation

Depth Pro was introduced in Depth Pro: Sharp Monocular Metric Depth in Less Than a Second, by Aleksei Bochkovskii, Amaël Delaunoy, Hugo Germain, Marcel Santos, Yichao Zhou, Stephan R. Richter, and Vladlen Koltun.

The checkpoint in this repository is a reference implementation, which has been re - trained. Its performance is close to the model reported in the paper but does not match it exactly.

📄 License

The license for this project is apple - amlr.

📦 Information

📄 Citation

If you find our work useful, please cite the following paper:

@article{Bochkovskii2024:arxiv,
  author     = {Aleksei Bochkovskii and Ama\"{e}l Delaunoy and Hugo Germain and Marcel Santos and
               Yichao Zhou and Stephan R. Richter and Vladlen Koltun}
  title      = {Depth Pro: Sharp Monocular Metric Depth in Less Than a Second},
  journal    = {arXiv},
  year       = {2024},
}

👏 Acknowledgements

Our codebase is built using multiple opensource contributions. Please see Acknowledgements for more details. Please check the paper for a complete list of references and datasets used in this work.