๐ Controlnet - v1.1 - depth Version
Controlnet v1.1 is a powerful model that enhances the capabilities of diffusion models. It allows for conditional control, enabling users to generate images based on specific input conditions, such as depth images. This model is a significant advancement in the field of text-to-image generation, providing more precise control over the output.
๐ Quick Start
Controlnet v1.1 is the successor model of Controlnet v1.0 and was released in lllyasviel/ControlNet-v1-1 by Lvmin Zhang.
This checkpoint is a conversion of the original checkpoint into diffusers
format. It can be used in combination with Stable Diffusion, such as runwayml/stable-diffusion-v1-5.
For more details, please also have a look at the ๐งจ Diffusers docs.
ControlNet is a neural network structure to control diffusion models by adding extra conditions.

This checkpoint corresponds to the ControlNet conditioned on depth images.
โจ Features
- Conditional Control: ControlNet allows diffusion models to be controlled by adding extra conditions, such as depth images, edge maps, and keypoints.
- End-to-End Learning: The model learns task-specific conditions in an end-to-end way, making it robust even with small training datasets.
- Fast Training: Training a ControlNet is as fast as fine-tuning a diffusion model and can be done on personal devices.
๐ฆ Installation
- Let's install
diffusers
and related packages:
$ pip install diffusers transformers accelerate
๐ป Usage Examples
Basic Usage
import torch
import os
from huggingface_hub import HfApi
from pathlib import Path
from diffusers.utils import load_image
from PIL import Image
import numpy as np
from transformers import pipeline
from diffusers import (
ControlNetModel,
StableDiffusionControlNetPipeline,
UniPCMultistepScheduler,
)
checkpoint = "lllyasviel/control_v11p_sd15_depth"
image = load_image(
"https://huggingface.co/lllyasviel/control_v11p_sd15_depth/resolve/main/images/input.png"
)
prompt = "Stormtrooper's lecture in beautiful lecture hall"
depth_estimator = pipeline('depth-estimation')
image = depth_estimator(image)['depth']
image = np.array(image)
image = image[:, :, None]
image = np.concatenate([image, image, image], axis=2)
control_image = Image.fromarray(image)
control_image.save("./images/control.png")
controlnet = ControlNetModel.from_pretrained(checkpoint, torch_dtype=torch.float16)
pipe = StableDiffusionControlNetPipeline.from_pretrained(
"runwayml/stable-diffusion-v1-5", controlnet=controlnet, torch_dtype=torch.float16
)
pipe.scheduler = UniPCMultistepScheduler.from_config(pipe.scheduler.config)
pipe.enable_model_cpu_offload()
generator = torch.manual_seed(0)
image = pipe(prompt, num_inference_steps=30, generator=generator, image=control_image).images[0]
image.save('images/image_out.png')



๐ Documentation
Model Details
Introduction
Controlnet was proposed in Adding Conditional Control to Text-to-Image Diffusion Models by Lvmin Zhang, Maneesh Agrawala.
The abstract reads as follows:
We present a neural network structure, ControlNet, to control pretrained large diffusion models to support additional input conditions. The ControlNet learns task-specific conditions in an end-to-end way, and the learning is robust even when the training dataset is small (< 50k). Moreover, training a ControlNet is as fast as fine-tuning a diffusion model, and the model can be trained on a personal devices. Alternatively, if powerful computation clusters are available, the model can scale to large amounts (millions to billions) of data. We report that large diffusion models like Stable Diffusion can be augmented with ControlNets to enable conditional inputs like edge maps, depthmentation maps, keypoints, etc. This may enrich the methods to control large diffusion models and further facilitate related applications.
Other released checkpoints v1-1
The authors released 14 different checkpoints, each trained with Stable Diffusion v1-5 on a different type of conditioning:
๐ License
The model is released under The CreativeML OpenRAIL M license, which is an Open RAIL M license, adapted from the work that BigScience and the RAIL Initiative are jointly carrying in the area of responsible AI licensing. See also the article about the BLOOM Open RAIL license on which our license is based.