FLUX.1 Dev ControlNet Union Pro 2.0

🚀 FLUX.1-dev-ControlNet-Union-Pro-2.0

This repository offers an unified ControlNet tailored for the FLUX.1-dev model, which is released by Shakker Labs. We've set up an online demo for easy testing. Additionally, a community-provided FP8 quantized version can be accessed at ABDALLALSWAITI/FLUX.1-dev-ControlNet-Union-Pro-2.0-fp8.

✨ Features

When compared with Shakker-Labs/FLUX.1-dev-ControlNet-Union-Pro:

• The mode embedding has been removed, resulting in a smaller model size.
• There are improvements in canny and pose, offering better control and aesthetics.
• Support for soft edge has been added, while support for tile has been removed.

📦 Installation

No installation steps are provided in the original document.

💻 Usage Examples

Basic Usage

import torch
from diffusers.utils import load_image
from diffusers import FluxControlNetPipeline, FluxControlNetModel

base_model = 'black-forest-labs/FLUX.1-dev'
controlnet_model_union = 'Shakker-Labs/FLUX.1-dev-ControlNet-Union-Pro-2.0'

controlnet = FluxControlNetModel.from_pretrained(controlnet_model_union, torch_dtype=torch.bfloat16)
pipe = FluxControlNetPipeline.from_pretrained(base_model, controlnet=controlnet, torch_dtype=torch.bfloat16)
pipe.to("cuda")

# replace with other conds
control_image = load_image("./conds/canny.png")
width, height = control_image.size

prompt = "A young girl stands gracefully at the edge of a serene beach, her long, flowing hair gently tousled by the sea breeze. She wears a soft, pastel-colored dress that complements the tranquil blues and greens of the coastal scenery. The golden hues of the setting sun cast a warm glow on her face, highlighting her serene expression. The background features a vast, azure ocean with gentle waves lapping at the shore, surrounded by distant cliffs and a clear, cloudless sky. The composition emphasizes the girl's serene presence amidst the natural beauty, with a balanced blend of warm and cool tones."

image = pipe(
    prompt, 
    control_image=control_image,
    width=width,
    height=height,
    controlnet_conditioning_scale=0.7,
    control_guidance_end=0.8,
    num_inference_steps=30, 
    guidance_scale=3.5,
    generator=torch.Generator(device="cuda").manual_seed(42),
).images[0]

Advanced Usage

import torch
from diffusers.utils import load_image

# https://github.com/huggingface/diffusers/pull/11350
# You can directly import from diffusers by install the laster version from source
# from diffusers import FluxControlNetPipeline, FluxControlNetModel

# use local files for this moment
from pipeline_flux_controlnet import FluxControlNetPipeline
from controlnet_flux import FluxControlNetModel

base_model = 'black-forest-labs/FLUX.1-dev'
controlnet_model_union = 'Shakker-Labs/FLUX.1-dev-ControlNet-Union-Pro-2.0'

controlnet = FluxControlNetModel.from_pretrained(controlnet_model_union, torch_dtype=torch.bfloat16)
pipe = FluxControlNetPipeline.from_pretrained(base_model, controlnet=[controlnet], torch_dtype=torch.bfloat16) # use [] to enable multi-CNs
pipe.to("cuda")

# replace with other conds
control_image = load_image("./conds/canny.png")
width, height = control_image.size

prompt = "A young girl stands gracefully at the edge of a serene beach, her long, flowing hair gently tousled by the sea breeze. She wears a soft, pastel-colored dress that complements the tranquil blues and greens of the coastal scenery. The golden hues of the setting sun cast a warm glow on her face, highlighting her serene expression. The background features a vast, azure ocean with gentle waves lapping at the shore, surrounded by distant cliffs and a clear, cloudless sky. The composition emphasizes the girl's serene presence amidst the natural beauty, with a balanced blend of warm and cool tones."

image = pipe(
    prompt, 
    control_image=[control_image, control_image], # try with different conds such as canny&depth, pose&depth
    width=width,
    height=height,
    controlnet_conditioning_scale=[0.35, 0.35],
    control_guidance_end=[0.8, 0.8],
    num_inference_steps=30, 
    guidance_scale=3.5,
    generator=torch.Generator(device="cuda").manual_seed(42),
).images[0]

📚 Documentation

Model Cards

• This ControlNet is composed of 6 double blocks and 0 single block, with the mode embedding removed.
• The model was trained from scratch for 300k steps using a dataset of 20M high - quality general and human images. Training was conducted at a 512x512 resolution in BFloat16, with a batch size of 128, a learning rate of 2e - 5, and the guidance uniformly sampled from [1, 7]. The text drop ratio was set to 0.20.
• This model supports multiple control modes, including canny, soft edge, depth, pose, and gray. It can be used just like a normal ControlNet.
• This model can be used in conjunction with other ControlNets.

Recommended Parameters

You can adjust controlnet_conditioning_scale and control_guidance_end for stronger control and better detail preservation. For better stability, we highly recommend using detailed prompts. In some cases, multi - conditions can be helpful.

• Canny: Use cv2.Canny, set controlnet_conditioning_scale = 0.7 and control_guidance_end = 0.8.
• Soft Edge: Use AnylineDetector, set controlnet_conditioning_scale = 0.7 and control_guidance_end = 0.8.
• Depth: Use depth - anything, set controlnet_conditioning_scale = 0.8 and control_guidance_end = 0.8.
• Pose: Use DWPose, set controlnet_conditioning_scale = 0.9 and control_guidance_end = 0.65.
• Gray: Use cv2.cvtColor, set controlnet_conditioning_scale = 0.9 and control_guidance_end = 0.8.

Resources

• InstantX/FLUX.1-dev-IP-Adapter
• InstantX/FLUX.1-dev-Controlnet-Canny
• Shakker-Labs/FLUX.1-dev-ControlNet-Depth
• Shakker-Labs/FLUX.1-dev-ControlNet-Union-Pro

🔧 Technical Details

No additional technical details are provided in the original document.

📄 License

The model is released under the flux-1-dev-non-commercial-license.

🎨 Showcases

Showcase	Image
Canny
Soft Edge
Pose
Depth
Gray

🙏 Acknowledgements

This model is developed by Shakker Labs. The original idea is inspired by xinsir/controlnet-union-sdxl-1.0. All copyright reserved.

📖 Citation

If you find this project useful in your research, please cite us via

@misc{flux-cn-union-pro-2,
    author = {Shakker-Labs},
    title = {ControlNet-Union},
    year = {2025},
    howpublished={\url{https://huggingface.co/Shakker-Labs/FLUX.1-dev-ControlNet-Union-Pro-2.0}},
}