Stable Diffusion 3.5 Large Controlnet Depth

🚀 Stable Diffusion 3.5 Large Controlnet - Depth

This repository offers the Depth ControlNet for Stable Diffusion 3.5 Large, enabling enhanced image generation with depth control.

🚀 Quick Start

This repository provides the Depth ControlNet for Stable Diffusion 3.5 Large..

Please note: This model is released under the Stability Community License. Visit Stability AI to learn or contact us for commercial licensing details.

✨ Features

• Flexible Usage: Can be used in both the SD3.5 Standalone Repo and the Diffusers library.
• Preprocessing Support: Provides preprocessing code snippets for input images.
• Safety Measures: Implements multiple safety evaluations and mitigations to prevent misuse.

📦 Installation

Using Controlnets in SD3.5 Standalone Repo

Install the repo:

git clone git@github.com:Stability-AI/sd3.5.git
pip install -r requirements.txt

Then, download the models and sample image like so:

input/sample_cond.png
models/clip_g.safetensors
models/clip_l.safetensors
models/t5xxl.safetensors
models/sd3.5_large.safetensors
models/canny_8b.safetensors

💻 Usage Examples

Basic Usage

Using Controlnets in SD3.5 Standalone Repo

python sd3_infer.py --controlnet_ckpt models/depth_8b.safetensors --controlnet_cond_image input/sample_cond.png --prompt "A girl sitting in a cafe, cozy interior, HDR photograph"

This command should give you an image like below:

Using Controlnets in Diffusers

import torch
from diffusers import StableDiffusion3ControlNetPipeline, SD3ControlNetModel
from diffusers.utils import load_image

controlnet = SD3ControlNetModel.from_pretrained("stabilityai/stable-diffusion-3.5-large-controlnet-depth", torch_dtype=torch.float16)
pipe = StableDiffusion3ControlNetPipeline.from_pretrained(
    "stabilityai/stable-diffusion-3.5-large",
    controlnet=controlnet,
    torch_dtype=torch.float16,
).to("cuda")

control_image = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/marigold/marigold_einstein_lcm_depth.png")
generator = torch.Generator(device="cpu").manual_seed(0)
image = pipe(
    prompt = "a photo of a man", 
    control_image=control_image, 
    guidance_scale=4.5,
    num_inference_steps=40,
    generator=generator,
    max_sequence_length=77,
).images[0]
image.save('depth-8b.jpg')

Advanced Usage

Extracting depth_image with image_gen_aux

# install image_gen_aux with: pip install git+https://github.com/huggingface/image_gen_aux.git
from image_gen_aux import DepthPreprocessor
image = load_image("path to image")

depth_preprocessor = DepthPreprocessor.from_pretrained("depth-anything/Depth-Anything-V2-Large-hf").to("cuda")
depth_image = depth_preprocessor(image, invert=True)[0].convert("RGB")

Preprocessing Input Images

# install depthfm from https://github.com/CompVis/depth-fm
import torchvision.transforms as transforms
from depthfm.dfm import DepthFM
depthfm_model = DepthFM(ckpt_path=checkpoint_path)
depthfm_model.eval()

# assuming img is a PIL image
img = F.to_tensor(img)
c, h, w = img.shape
img = F.interpolate(img, (512, 512), mode='bilinear', align_corners=False)
with torch.no_grad():
  img = self.depthfm_model(img, num_steps=2, ensemble_size=4)
img = F.interpolate(img, (h, w), mode='bilinear', align_corners=False)

Tips

⚠️ Important Note

All uses of the model must be in accordance with our Acceptable Use Policy.

💡 Usage Tip

• We recommend starting with a ControlNet strength of 0.7, and adjusting as needed.
• Euler sampler and a slightly higher step count (50 - 60) gives best results.
• Pass --text_encoder_device <device_name> to load the text encoders directly to VRAM, which can speed up the full inference loop at the cost of extra VRAM usage.

📚 Documentation

Uses

All uses of the model must be in accordance with our Acceptable Use Policy.

Out-of-Scope Uses

The model was not trained to be factual or true representations of people or events. As such, using the model to generate such content is out-of-scope of the abilities of this model.

Training Data and Strategy

These models were trained on a wide variety of data, including synthetic data and filtered publicly available data.

Safety

We believe in safe, responsible AI practices and take deliberate measures to ensure Integrity starts at the early stages of development. This means we have taken and continue to take reasonable steps to prevent the misuse of Stable Diffusion 3.5 by bad actors. For more information about our approach to Safety please visit our Safety page.

Integrity Evaluation

Our integrity evaluation methods include structured evaluations and red-teaming testing for certain harms. Testing was conducted primarily in English and may not cover all possible harms.

Risks identified and mitigations:

Acknowledgements

• Lvmin Zhang, Anyi Rao, and Maneesh Agrawala, authors of the original ControlNet paper.
• Lvmin Zhang, who also developed the Tile ControlNet, which inspired the Blur ControlNet.
• Diffusers library authors, whose code was referenced during development.
• InstantX team, whose Flux and SD3 ControlNets were also referenced during training.
• All early testers and raters of the models, and the Stability AI team.

Contact

Please report any issues with the model or contact us:

• Safety issues: safety@stability.ai
• Security issues: security@stability.ai
• Privacy issues: privacy@stability.ai
• License and general: https://stability.ai/license
• Enterprise license: https://stability.ai/enterprise

📄 License

Here are the key components of the license:

• Free for non-commercial use: Individuals and organizations can use the model free of charge for non-commercial use, including scientific research.
• Free for commercial use (up to $1M in annual revenue): Startups, small to medium-sized businesses, and creators can use the model for commercial purposes at no cost, as long as their total annual revenue is less than $1M.
• Ownership of outputs: Retain ownership of the media generated without restrictive licensing implications.

For organizations with annual revenue more than $1M, please contact us here to inquire about an Enterprise License.