t2iadapter_depth_sd14v1 Open Source Model - Provides Depth Conditional Control Functionality for Stable Diffusion 1.4

T2iadapter Depth Sd14v1

Developed by TencentARC

T2I Adapter is a network that provides additional conditional control for Stable Diffusion models, with the depth version specifically offering depth-based conditional control for Stable Diffusion v1.4.

Image Generation OtherOpen Source License:Apache-2.0 #Depth Map Controlled Generation #StableDiffusion Adaptation #Image Structure Control

Downloads 14

Release Time : 7/14/2023

Model Overview

This model enhances the spatial structure and hierarchical control capabilities of Stable Diffusion during image generation by using depth maps as conditional inputs.

Model Features

Depth-Based Conditional Control

Precisely controls the 3D structure and spatial hierarchy of generated images using grayscale depth maps (black represents deep areas, white represents shallow areas).

Lightweight Adapter Design

Employs an adapter network independent of the base model, enhancing control capabilities without modifying the original model.

Multi-Condition Compatibility

Can be combined with text prompts to achieve dual control over semantics and geometric structures.

Model Capabilities

Depth-Conditioned Image Generation

Structure-Preserving Image Synthesis

Controllable 3D Scene Generation

Use Cases

Digital Art Creation

Architectural Concept Design

Generates architectural concept images with precise perspective structures based on depth maps and text descriptions.

Artistic renderings that maintain the original depth structure.

Game Asset Generation

Scene Element Generation

Controls the spatial layout of objects in game scenes using depth maps.

Hierarchical scene generation that meets design requirements.

🚀 T2I Adapter - Depth

T2I Adapter is a network that provides additional conditioning to stable diffusion. Each T2I checkpoint takes a different type of conditioning as input and is used with a specific base stable diffusion checkpoint. This checkpoint offers depth conditioning for the stable diffusion 1.4 checkpoint.

🚀 Quick Start

Dependencies

pip install diffusers transformers controlnet_aux

Run code

from controlnet_aux import MidasDetector
from PIL import Image
from diffusers import T2IAdapter, StableDiffusionAdapterPipeline
import torch

midas = MidasDetector.from_pretrained("lllyasviel/Annotators")

image = Image.open('./images/depth_input.png')

image = midas(image)

image.save('./images/depth.png')

adapter = T2IAdapter.from_pretrained("TencentARC/t2iadapter_depth_sd14v1", torch_dtype=torch.float16)
pipe = StableDiffusionAdapterPipeline.from_pretrained(
    "CompVis/stable-diffusion-v1-4", adapter=adapter, safety_checker=None, torch_dtype=torch.float16, variant="fp16"
)

pipe.to('cuda')

generator = torch.Generator().manual_seed(1)

openpose_out = pipe(prompt="storm trooper giving a speech", image=image, generator=generator).images[0]

openpose_out.save('./images/depth_output.png')

depth_input depth depth_output

✨ Features

T2I Adapter provides additional conditioning to stable diffusion, enabling more controllable text - to - image generation. Different checkpoints can take various types of conditioning as input, enhancing the model's flexibility.

📚 Documentation

Model Details

Property	Details
Developed by	T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models
Model Type	Diffusion-based text-to-image generation model
Language(s)	English
License	Apache 2.0
Resources for more information	GitHub Repository, Paper.
Cite as	@misc{ title={T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models}, author={Chong Mou, Xintao Wang, Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi, Ying Shan, Xiaohu Qie}, year={2023}, eprint={2302.08453}, archivePrefix={arXiv}, primaryClass={cs.CV} }

Checkpoints

Model Name	Control Image Overview	Control Image Example	Generated Image Example
TencentARC/t2iadapter_color_sd14v1 Trained with spatial color palette	A image with 8x8 color palette.
TencentARC/t2iadapter_canny_sd14v1 Trained with canny edge detection	A monochrome image with white edges on a black background.
TencentARC/t2iadapter_sketch_sd14v1 Trained with PidiNet edge detection	A hand-drawn monochrome image with white outlines on a black background.
TencentARC/t2iadapter_depth_sd14v1 Trained with Midas depth estimation	A grayscale image with black representing deep areas and white representing shallow areas.
TencentARC/t2iadapter_openpose_sd14v1 Trained with OpenPose bone image	A OpenPose bone image.
TencentARC/t2iadapter_keypose_sd14v1 Trained with mmpose skeleton image	A mmpose skeleton image.
TencentARC/t2iadapter_seg_sd14v1 Trained with semantic segmentation	An custom segmentation protocol image.
TencentARC/t2iadapter_canny_sd15v2
TencentARC/t2iadapter_depth_sd15v2
TencentARC/t2iadapter_sketch_sd15v2
TencentARC/t2iadapter_zoedepth_sd15v1

📄 License

This project is licensed under the Apache 2.0 license.

Featured Recommended AI Models

Empowering the Future, Your AI Solution Knowledge Base

English 简体中文繁體中文にほんご