Controlnet Openpose Sdxl 1.0

🚀 SDXL-controlnet: OpenPose (v2)

These controlnet weights are trained on stabilityai/stable-diffusion-xl-base-1.0 with OpenPose (v2) conditioning, enabling text-to-image generation with specific pose control.

🚀 Quick Start

These are controlnet weights trained on stabilityai/stable-diffusion-xl-base-1.0 with OpenPose (v2) conditioning. You can find some example images below.

Prompt: a ballerina, romantic sunset, 4k photo

Comfy Workflow

(The image is from ComfyUI. You can drag and drop it in Comfy to use it as a workflow.)

License: refers to the OpenPose's one.

✨ Features

• Pose - Controlled Image Generation: Leverage OpenPose (v2) conditioning to generate images with specific poses.
• Integration with Diffusers: Seamlessly work with the 🧨 diffusers library.

📦 Installation

First, install all the libraries:

pip install -q controlnet_aux transformers accelerate
pip install -q git+https://github.com/huggingface/diffusers

💻 Usage Examples

Basic Usage

from diffusers import AutoencoderKL, StableDiffusionXLControlNetPipeline, ControlNetModel, UniPCMultistepScheduler
import torch
from controlnet_aux import OpenposeDetector
from diffusers.utils import load_image

# Compute openpose conditioning image.
openpose = OpenposeDetector.from_pretrained("lllyasviel/ControlNet")

image = load_image(
    "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/person.png"
)
openpose_image = openpose(image)

# Initialize ControlNet pipeline.
controlnet = ControlNetModel.from_pretrained("thibaud/controlnet-openpose-sdxl-1.0", torch_dtype=torch.float16)
pipe = StableDiffusionXLControlNetPipeline.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0", controlnet=controlnet, torch_dtype=torch.float16
)
pipe.enable_model_cpu_offload()

# Infer.
prompt = "Darth vader dancing in a desert, high quality"
negative_prompt = "low quality, bad quality"
images = pipe(
    prompt, 
    negative_prompt=negative_prompt,
    num_inference_steps=25,
    num_images_per_prompt=4,
    image=openpose_image.resize((1024, 1024)),
    generator=torch.manual_seed(97),
).images
images[0]

Here are some generated examples:

🔧 Technical Details

Training

Use the training script by HF🤗 here.

Training Data

This checkpoint was first trained for 15,000 steps on laion 6a resized to a max minimum dimension of 768.

Compute

One 1xA100 machine (Thanks a lot HF🤗 for providing the compute!)

Batch Size

Data parallel with a single gpu batch size of 2 with gradient accumulation 8.

Hyper Parameters

Constant learning rate of 8e-5

Mixed Precision

fp16

📄 License

Refers to the OpenPose's one.