t2iadapter_depth_sd15v2 Open-source Model - Providing Depth Condition Control for Stable Diffusion 1.5 Version

T2iadapter Depth Sd15v2

Developed by TencentARC

T2I Adapter is a network that provides additional conditional control for stable diffusion models. This checkpoint is specifically designed to provide depth conditional control for Stable Diffusion 1.5.

Image Generation OtherOpen Source License:Apache-2.0 #Depth Map Controlled Generation #Stable Diffusion Adaptation #3D Scene Reconstruction

Downloads 2,217

Release Time : 7/14/2023

Model Overview

A diffusion-based text-to-image generation model that controls image generation through depth map conditions, suitable for tasks requiring precise spatial structure in image generation.

Model Features

Depth Condition Control

Precisely controls the spatial structure and hierarchical relationships of generated images through grayscale depth maps.

Lightweight Adapter Design

As an independent network module, it enhances control capabilities without modifying the base model.

Multi-Condition Support

Can be combined with text prompts to achieve multimodal conditional control.

Model Capabilities

Depth-Conditioned Image Generation

Text-to-Image Conversion

Spatial Structure Control

Image Synthesis

Use Cases

Creative Design

Concept Scene Generation

Generate scene images with precise spatial layouts based on depth maps and text descriptions.

Maintains the spatial structure of the input depth map while enabling creative expression.

Film & Gaming

Scene Prototype Design

Quickly generate scene concept images that meet specific spatial requirements.

Accelerates the early-stage concept design process.

🚀 T2I Adapter - Depth

T2I Adapter is a network that provides additional conditioning to stable diffusion. Each T2I checkpoint takes a different type of conditioning as input and is used with a specific base stable diffusion checkpoint. This checkpoint offers depth conditioning for the Stable Diffusion 1.5 checkpoint.

✨ Features

Enables additional conditioning for Stable Diffusion.
Supports different types of conditioning inputs.
Specifically provides depth conditioning for Stable Diffusion 1.5.

📦 Installation

Dependencies

pip install diffusers transformers controlnet_aux

💻 Usage Examples

Basic Usage

from controlnet_aux import MidasDetector
from PIL import Image
from diffusers import T2IAdapter, StableDiffusionAdapterPipeline
import torch

midas = MidasDetector.from_pretrained("lllyasviel/Annotators")

image = Image.open('./images/depth_input.png')

image = midas(image)

image.save('./images/depth.png')

adapter = T2IAdapter.from_pretrained("TencentARC/t2iadapter_depth_sd15v2", torch_dtype=torch.float16)
pipe = StableDiffusionAdapterPipeline.from_pretrained(
    "runwayml/stable-diffusion-v1-5", adapter=adapter, safety_checker=None, torch_dtype=torch.float16, variant="fp16"
)

pipe.to('cuda')

generator = torch.Generator().manual_seed(1)

depth_out = pipe(prompt="storm trooper giving a speech", image=image, generator=generator).images[0]

depth_out.save('./images/depth_output.png')

Example Images

depth_input depth depth_output

📚 Documentation

Model Details

Property	Details
Developed by	T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models
Model Type	Diffusion-based text-to-image generation model
Language(s)	English
License	Apache 2.0
Resources for more information	GitHub Repository, Paper
Cite as	@misc{ title={T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models}, author={Chong Mou, Xintao Wang, Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi, Ying Shan, Xiaohu Qie}, year={2023}, eprint={2302.08453}, archivePrefix={arXiv}, primaryClass={cs.CV} }

Checkpoints

Model Name	Control Image Overview	Control Image Example	Generated Image Example
TencentARC/t2iadapter_color_sd14v1 Trained with spatial color palette	A image with 8x8 color palette.
TencentARC/t2iadapter_canny_sd14v1 Trained with canny edge detection	A monochrome image with white edges on a black background.
TencentARC/t2iadapter_sketch_sd14v1 Trained with PidiNet edge detection	A hand-drawn monochrome image with white outlines on a black background.
TencentARC/t2iadapter_depth_sd14v1 Trained with Midas depth estimation	A grayscale image with black representing deep areas and white representing shallow areas.
TencentARC/t2iadapter_openpose_sd14v1 Trained with OpenPose bone image	A OpenPose bone image.
TencentARC/t2iadapter_keypose_sd14v1 Trained with mmpose skeleton image	A mmpose skeleton image.
TencentARC/t2iadapter_seg_sd14v1 Trained with semantic segmentation	An custom segmentation protocol image.
TencentARC/t2iadapter_canny_sd15v2
TencentARC/t2iadapter_depth_sd15v2
TencentARC/t2iadapter_sketch_sd15v2
TencentARC/t2iadapter_zoedepth_sd15v1

📄 License

This project is licensed under the Apache 2.0 license.

Featured Recommended AI Models

Empowering the Future, Your AI Solution Knowledge Base

English 简体中文繁體中文にほんご