Controlnet Canny Sdxl 1.0

🚀 SDXL-controlnet: Canny

Controlnet weights trained on stabilityai/stable-diffusion-xl-base-1.0 with canny conditioning, enabling high - quality text - to - image generation.

📦 Information Table

🖼️ Example Images

Here are some example images generated with different prompts:

• Prompt: a couple watching a romantic sunset, 4k photo
• Prompt: ultrarealistic shot of a furry blue bird
• Prompt: a woman, close up, detailed, beautiful, street photography, photorealistic, detailed, Kodak ektar 100, natural, candid shot
• Prompt: Cinematic, neoclassical table in the living room, cinematic, contour, lighting, highly detailed, winter, golden hour
• Prompt: a tornado hitting grass field, 1980's film grain. overcast, muted colors.

🚀 Quick Start

📦 Installation

Make sure to first install the necessary libraries:

pip install accelerate transformers safetensors opencv-python diffusers

💻 Usage Examples

Basic Usage

from diffusers import ControlNetModel, StableDiffusionXLControlNetPipeline, AutoencoderKL
from diffusers.utils import load_image
from PIL import Image
import torch
import numpy as np
import cv2

prompt = "aerial view, a futuristic research complex in a bright foggy jungle, hard lighting"
negative_prompt = 'low quality, bad quality, sketches'

image = load_image("https://huggingface.co/datasets/hf-internal-testing/diffusers-images/resolve/main/sd_controlnet/hf-logo.png")

controlnet_conditioning_scale = 0.5  # recommended for good generalization

controlnet = ControlNetModel.from_pretrained(
    "diffusers/controlnet-canny-sdxl-1.0",
    torch_dtype=torch.float16
)
vae = AutoencoderKL.from_pretrained("madebyollin/sdxl-vae-fp16-fix", torch_dtype=torch.float16)
pipe = StableDiffusionXLControlNetPipeline.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0",
    controlnet=controlnet,
    vae=vae,
    torch_dtype=torch.float16,
)
pipe.enable_model_cpu_offload()

image = np.array(image)
image = cv2.Canny(image, 100, 200)
image = image[:, :, None]
image = np.concatenate([image, image, image], axis=2)
image = Image.fromarray(image)

images = pipe(
    prompt, negative_prompt=negative_prompt, image=image, controlnet_conditioning_scale=controlnet_conditioning_scale,
    ).images

images[0].save(f"hug_lab.png")

For more details, check out the official documentation of StableDiffusionXLControlNetPipeline.

🔧 Technical Details

Training

Our training script was built on top of the official training script provided here.

Training Data

This checkpoint was first trained for 20,000 steps on laion 6a resized to a max minimum dimension of 384. It was then further trained for 20,000 steps on laion 6a resized to a max minimum dimension of 1024 and then filtered to contain only minimum 1024 images. High - resolution finetuning was found necessary for image quality.

Compute

One 8xA100 machine was used for training.

Batch Size

Data parallel with a single gpu batch size of 8 for a total batch size of 64.

Hyper Parameters

A constant learning rate of 1e - 4 scaled by batch size for a total learning rate of 64e - 4.

Mixed Precision

fp16 was used for mixed precision training.