FLUX.1 Dev ControlNet Union Pro 2.0

🚀 FLUX.1-dev-ControlNet-Union-Pro-2.0

本倉庫包含由 Shakker Labs 發佈的用於 FLUX.1-dev 模型的統一 ControlNet。我們提供了一個 在線演示。社區提供的 FP8 量化版本可在 ABDALLALSWAITI/FLUX.1-dev-ControlNet-Union-Pro-2.0-fp8 中找到。

✨ 主要特性

與 Shakker-Labs/FLUX.1-dev-ControlNet-Union-Pro 相比：

• 移除了模式嵌入，模型尺寸更小。
• 在 Canny 和姿態控制方面有所改進，控制效果和美學表現更佳。
• 增加了對軟邊緣的支持，移除了對平鋪模式的支持。

📚 詳細文檔

模型卡片

• 此 ControlNet 由 6 個雙塊和 0 個單塊組成，移除了模式嵌入。
• 我們使用包含 2000 萬張高質量通用和人物圖像的數據集，從零開始對模型進行了 30 萬步的訓練。訓練分辨率為 512x512，使用 BFloat16 格式，批量大小為 128，學習率為 2e-5，引導係數從 [1, 7] 中均勻採樣。我們將文本丟棄率設置為 0.20。
• 該模型支持多種控制模式，包括 Canny、軟邊緣、深度、姿態和灰度。您可以像使用普通的 ControlNet 一樣使用它。
• 此模型可以與其他 ControlNet 聯合使用。

展示示例

控制模式	示例圖片
Canny
軟邊緣
姿態
深度
灰度

💻 使用示例

基礎用法

import torch
from diffusers.utils import load_image
from diffusers import FluxControlNetModel, FluxControlNetPipeline

base_model = 'black-forest-labs/FLUX.1-dev'
controlnet_model_union = 'Shakker-Labs/FLUX.1-dev-ControlNet-Union-Pro-2.0'

controlnet = FluxControlNetModel.from_pretrained(controlnet_model_union, torch_dtype=torch.bfloat16)
pipe = FluxControlNetPipeline.from_pretrained(base_model, controlnet=controlnet, torch_dtype=torch.bfloat16)
pipe.to("cuda")

# 替換為其他條件圖像
control_image = load_image("./conds/canny.png")
width, height = control_image.size

prompt = "A young girl stands gracefully at the edge of a serene beach, her long, flowing hair gently tousled by the sea breeze. She wears a soft, pastel-colored dress that complements the tranquil blues and greens of the coastal scenery. The golden hues of the setting sun cast a warm glow on her face, highlighting her serene expression. The background features a vast, azure ocean with gentle waves lapping at the shore, surrounded by distant cliffs and a clear, cloudless sky. The composition emphasizes the girl's serene presence amidst the natural beauty, with a balanced blend of warm and cool tones."

image = pipe(
    prompt, 
    control_image=control_image,
    width=width,
    height=height,
    controlnet_conditioning_scale=0.7,
    control_guidance_end=0.8,
    num_inference_steps=30, 
    guidance_scale=3.5,
    generator=torch.Generator(device="cuda").manual_seed(42),
).images[0]

高級用法

import torch
from diffusers.utils import load_image

# https://github.com/huggingface/diffusers/pull/11350
# 您可以通過從源代碼安裝最新版本直接從 diffusers 導入
# from diffusers import FluxControlNetPipeline, FluxControlNetModel

# 目前使用本地文件
from pipeline_flux_controlnet import FluxControlNetPipeline
from controlnet_flux import FluxControlNetModel

base_model = 'black-forest-labs/FLUX.1-dev'
controlnet_model_union = 'Shakker-Labs/FLUX.1-dev-ControlNet-Union-Pro-2.0'

controlnet = FluxControlNetModel.from_pretrained(controlnet_model_union, torch_dtype=torch.bfloat16)
pipe = FluxControlNetPipeline.from_pretrained(base_model, controlnet=[controlnet], torch_dtype=torch.bfloat16) # 使用列表以啟用多 ControlNet
pipe.to("cuda")

# 替換為其他條件圖像
control_image = load_image("./conds/canny.png")
width, height = control_image.size

prompt = "A young girl stands gracefully at the edge of a serene beach, her long, flowing hair gently tousled by the sea breeze. She wears a soft, pastel-colored dress that complements the tranquil blues and greens of the coastal scenery. The golden hues of the setting sun cast a warm glow on her face, highlighting her serene expression. The background features a vast, azure ocean with gentle waves lapping at the shore, surrounded by distant cliffs and a clear, cloudless sky. The composition emphasizes the girl's serene presence amidst the natural beauty, with a balanced blend of warm and cool tones."

image = pipe(
    prompt, 
    control_image=[control_image, control_image], # 嘗試使用不同的條件，如 Canny 和深度、姿態和深度
    width=width,
    height=height,
    controlnet_conditioning_scale=[0.35, 0.35],
    control_guidance_end=[0.8, 0.8],
    num_inference_steps=30, 
    guidance_scale=3.5,
    generator=torch.Generator(device="cuda").manual_seed(42),
).images[0]

推薦參數

您可以調整 controlnet_conditioning_scale 和 control_guidance_end 以獲得更強的控制效果和更好的細節保留。為了獲得更好的穩定性，我們強烈建議使用詳細的提示詞，在某些情況下，多條件控制會有所幫助。

• Canny：使用 cv2.Canny，controlnet_conditioning_scale = 0.7，control_guidance_end = 0.8。
• 軟邊緣：使用 AnylineDetector，controlnet_conditioning_scale = 0.7，control_guidance_end = 0.8。
• 深度：使用 depth-anything，controlnet_conditioning_scale = 0.8，control_guidance_end = 0.8。
• 姿態：使用 DWPose，controlnet_conditioning_scale = 0.9，control_guidance_end = 0.65。
• 灰度：使用 cv2.cvtColor，controlnet_conditioning_scale = 0.9，control_guidance_end = 0.8。

📄 許可證

本項目使用 flux-1-dev-non-commercial-license 許可證。

🔗 相關資源

• InstantX/FLUX.1-dev-IP-Adapter
• InstantX/FLUX.1-dev-Controlnet-Canny
• Shakker-Labs/FLUX.1-dev-ControlNet-Depth
• Shakker-Labs/FLUX.1-dev-ControlNet-Union-Pro

🙏 致謝

本模型由 Shakker Labs 開發。最初的想法受到 xinsir/controlnet-union-sdxl-1.0 的啟發。保留所有權利。

📑 引用

如果您在研究中發現本項目有用，請通過以下方式引用我們：

@misc{flux-cn-union-pro-2,
    author = {Shakker-Labs},
    title = {ControlNet-Union},
    year = {2025},
    howpublished={\url{https://huggingface.co/Shakker-Labs/FLUX.1-dev-ControlNet-Union-Pro-2.0}},
}