T2i Adapter Depth Midas Sdxl 1.0

🚀 T2I-Adapter-SDXL - Depth-MiDaS

T2I Adapter is a network that offers additional conditioning for stable diffusion. Each t2i checkpoint takes a different type of conditioning as input and is used with a specific base stable diffusion checkpoint. This checkpoint provides depth conditioning for the StableDiffusionXL checkpoint. It's a collaboration between Tencent ARC and Hugging Face.

🚀 Quick Start

To get started, first install the required dependencies:

pip install -U git+https://github.com/huggingface/diffusers.git
pip install -U controlnet_aux==0.0.7 # for conditioning models and detectors
pip install transformers accelerate safetensors

1. Images are first downloaded into the appropriate control image format.
2. The control image and prompt are passed to the StableDiffusionXLAdapterPipeline.

✨ Features

T2I Adapter provides additional conditioning to stable diffusion, enabling more controllable text - to - image generation. Each checkpoint can take different types of conditioning as input, enhancing the flexibility and diversity of image generation.

📦 Installation

pip install -U git+https://github.com/huggingface/diffusers.git
pip install -U controlnet_aux==0.0.7 # for conditioning models and detectors
pip install transformers accelerate safetensors

💻 Usage Examples

Basic Usage

Let's have a look at a simple example using the Canny Adapter.

• Dependency

from diffusers import StableDiffusionXLAdapterPipeline, T2IAdapter, EulerAncestralDiscreteScheduler, AutoencoderKL
from diffusers.utils import load_image, make_image_grid
from controlnet_aux.midas import MidasDetector
import torch

# load adapter
adapter = T2IAdapter.from_pretrained(
  "TencentARC/t2i-adapter-depth-midas-sdxl-1.0", torch_dtype=torch.float16, varient="fp16"
).to("cuda")

# load euler_a scheduler
model_id = 'stabilityai/stable-diffusion-xl-base-1.0'
euler_a = EulerAncestralDiscreteScheduler.from_pretrained(model_id, subfolder="scheduler")
vae=AutoencoderKL.from_pretrained("madebyollin/sdxl-vae-fp16-fix", torch_dtype=torch.float16)
pipe = StableDiffusionXLAdapterPipeline.from_pretrained(
    model_id, vae=vae, adapter=adapter, scheduler=euler_a, torch_dtype=torch.float16, variant="fp16", 
).to("cuda")
pipe.enable_xformers_memory_efficient_attention()

midas_depth = MidasDetector.from_pretrained(
  "valhalla/t2iadapter-aux-models", filename="dpt_large_384.pt", model_type="dpt_large"
).to("cuda")

• Condition Image

url = "https://huggingface.co/Adapter/t2iadapter/resolve/main/figs_SDXLV1.0/org_mid.jpg"
image = load_image(url)
image = midas_depth(
  image, detect_resolution=512, image_resolution=1024
)

• Generation

prompt = "A photo of a room, 4k photo, highly detailed"
negative_prompt = "anime, cartoon, graphic, text, painting, crayon, graphite, abstract, glitch, deformed, mutated, ugly, disfigured"

gen_images = pipe(
  prompt=prompt,
  negative_prompt=negative_prompt,
  image=image,
  num_inference_steps=30,
  adapter_conditioning_scale=1,
  guidance_scale=7.5,  
).images[0]
gen_images.save('out_mid.png')

📚 Documentation

Model Details

Checkpoints

Model Name	Control Image Overview	Control Image Example	Generated Image Example
TencentARC/t2i-adapter-canny-sdxl-1.0 Trained with canny edge detection	A monochrome image with white edges on a black background.
TencentARC/t2i-adapter-sketch-sdxl-1.0 Trained with PidiNet edge detection	A hand-drawn monochrome image with white outlines on a black background.
TencentARC/t2i-adapter-lineart-sdxl-1.0 Trained with lineart edge detection	A hand-drawn monochrome image with white outlines on a black background.
TencentARC/t2i-adapter-depth-midas-sdxl-1.0 Trained with Midas depth estimation	A grayscale image with black representing deep areas and white representing shallow areas.
TencentARC/t2i-adapter-depth-zoe-sdxl-1.0 Trained with Zoe depth estimation	A grayscale image with black representing deep areas and white representing shallow areas.
TencentARC/t2i-adapter-openpose-sdxl-1.0 Trained with OpenPose bone image	A OpenPose bone image.

Training

Our training script was built on top of the official training script that we provide here.

The model is trained on 3M high-resolution image-text pairs from LAION-Aesthetics V2 with:

• Training steps: 35000
• Batch size: Data parallel with a single gpu batch size of 16 for a total batch size of 256.
• Learning rate: Constant learning rate of 1e-5.
• Mixed precision: fp16

📄 License

This model is licensed under the Apache 2.0 license.