FLUX.1 Dev ControlNet Union Pro 2.0

🚀 FLUX.1-dev-ControlNet-Union-Pro-2.0

本仓库包含由 Shakker Labs 发布的用于 FLUX.1-dev 模型的统一 ControlNet。我们提供了一个 在线演示。社区提供的 FP8 量化版本可在 ABDALLALSWAITI/FLUX.1-dev-ControlNet-Union-Pro-2.0-fp8 中找到。

✨ 主要特性

与 Shakker-Labs/FLUX.1-dev-ControlNet-Union-Pro 相比：

• 移除了模式嵌入，模型尺寸更小。
• 在 Canny 和姿态控制方面有所改进，控制效果和美学表现更佳。
• 增加了对软边缘的支持，移除了对平铺模式的支持。

📚 详细文档

模型卡片

• 此 ControlNet 由 6 个双块和 0 个单块组成，移除了模式嵌入。
• 我们使用包含 2000 万张高质量通用和人物图像的数据集，从零开始对模型进行了 30 万步的训练。训练分辨率为 512x512，使用 BFloat16 格式，批量大小为 128，学习率为 2e-5，引导系数从 [1, 7] 中均匀采样。我们将文本丢弃率设置为 0.20。
• 该模型支持多种控制模式，包括 Canny、软边缘、深度、姿态和灰度。您可以像使用普通的 ControlNet 一样使用它。
• 此模型可以与其他 ControlNet 联合使用。

展示示例

控制模式	示例图片
Canny
软边缘
姿态
深度
灰度

💻 使用示例

基础用法

import torch
from diffusers.utils import load_image
from diffusers import FluxControlNetModel, FluxControlNetPipeline

base_model = 'black-forest-labs/FLUX.1-dev'
controlnet_model_union = 'Shakker-Labs/FLUX.1-dev-ControlNet-Union-Pro-2.0'

controlnet = FluxControlNetModel.from_pretrained(controlnet_model_union, torch_dtype=torch.bfloat16)
pipe = FluxControlNetPipeline.from_pretrained(base_model, controlnet=controlnet, torch_dtype=torch.bfloat16)
pipe.to("cuda")

# 替换为其他条件图像
control_image = load_image("./conds/canny.png")
width, height = control_image.size

prompt = "A young girl stands gracefully at the edge of a serene beach, her long, flowing hair gently tousled by the sea breeze. She wears a soft, pastel-colored dress that complements the tranquil blues and greens of the coastal scenery. The golden hues of the setting sun cast a warm glow on her face, highlighting her serene expression. The background features a vast, azure ocean with gentle waves lapping at the shore, surrounded by distant cliffs and a clear, cloudless sky. The composition emphasizes the girl's serene presence amidst the natural beauty, with a balanced blend of warm and cool tones."

image = pipe(
    prompt, 
    control_image=control_image,
    width=width,
    height=height,
    controlnet_conditioning_scale=0.7,
    control_guidance_end=0.8,
    num_inference_steps=30, 
    guidance_scale=3.5,
    generator=torch.Generator(device="cuda").manual_seed(42),
).images[0]

高级用法

import torch
from diffusers.utils import load_image

# https://github.com/huggingface/diffusers/pull/11350
# 您可以通过从源代码安装最新版本直接从 diffusers 导入
# from diffusers import FluxControlNetPipeline, FluxControlNetModel

# 目前使用本地文件
from pipeline_flux_controlnet import FluxControlNetPipeline
from controlnet_flux import FluxControlNetModel

base_model = 'black-forest-labs/FLUX.1-dev'
controlnet_model_union = 'Shakker-Labs/FLUX.1-dev-ControlNet-Union-Pro-2.0'

controlnet = FluxControlNetModel.from_pretrained(controlnet_model_union, torch_dtype=torch.bfloat16)
pipe = FluxControlNetPipeline.from_pretrained(base_model, controlnet=[controlnet], torch_dtype=torch.bfloat16) # 使用列表以启用多 ControlNet
pipe.to("cuda")

# 替换为其他条件图像
control_image = load_image("./conds/canny.png")
width, height = control_image.size

prompt = "A young girl stands gracefully at the edge of a serene beach, her long, flowing hair gently tousled by the sea breeze. She wears a soft, pastel-colored dress that complements the tranquil blues and greens of the coastal scenery. The golden hues of the setting sun cast a warm glow on her face, highlighting her serene expression. The background features a vast, azure ocean with gentle waves lapping at the shore, surrounded by distant cliffs and a clear, cloudless sky. The composition emphasizes the girl's serene presence amidst the natural beauty, with a balanced blend of warm and cool tones."

image = pipe(
    prompt, 
    control_image=[control_image, control_image], # 尝试使用不同的条件，如 Canny 和深度、姿态和深度
    width=width,
    height=height,
    controlnet_conditioning_scale=[0.35, 0.35],
    control_guidance_end=[0.8, 0.8],
    num_inference_steps=30, 
    guidance_scale=3.5,
    generator=torch.Generator(device="cuda").manual_seed(42),
).images[0]

推荐参数

您可以调整 controlnet_conditioning_scale 和 control_guidance_end 以获得更强的控制效果和更好的细节保留。为了获得更好的稳定性，我们强烈建议使用详细的提示词，在某些情况下，多条件控制会有所帮助。

• Canny：使用 cv2.Canny，controlnet_conditioning_scale = 0.7，control_guidance_end = 0.8。
• 软边缘：使用 AnylineDetector，controlnet_conditioning_scale = 0.7，control_guidance_end = 0.8。
• 深度：使用 depth-anything，controlnet_conditioning_scale = 0.8，control_guidance_end = 0.8。
• 姿态：使用 DWPose，controlnet_conditioning_scale = 0.9，control_guidance_end = 0.65。
• 灰度：使用 cv2.cvtColor，controlnet_conditioning_scale = 0.9，control_guidance_end = 0.8。

📄 许可证

本项目使用 flux-1-dev-non-commercial-license 许可证。

🔗 相关资源

• InstantX/FLUX.1-dev-IP-Adapter
• InstantX/FLUX.1-dev-Controlnet-Canny
• Shakker-Labs/FLUX.1-dev-ControlNet-Depth
• Shakker-Labs/FLUX.1-dev-ControlNet-Union-Pro

🙏 致谢

本模型由 Shakker Labs 开发。最初的想法受到 xinsir/controlnet-union-sdxl-1.0 的启发。保留所有权利。

📑 引用

如果您在研究中发现本项目有用，请通过以下方式引用我们：

@misc{flux-cn-union-pro-2,
    author = {Shakker-Labs},
    title = {ControlNet-Union},
    year = {2025},
    howpublished={\url{https://huggingface.co/Shakker-Labs/FLUX.1-dev-ControlNet-Union-Pro-2.0}},
}