FLUX.1 Dev Edit V0

✨ 主要特性

• 基於特定模型和大規模數據集訓練，具備強大的圖像編輯能力。
• 支持使用Flux Control框架進行微調。
• 可通過不同參數設置（如 guidance_scale）和技術（如turbo LoRA、量化）優化推理效果和性能。

💻 使用示例

基礎用法

from diffusers import FluxControlPipeline, FluxTransformer2DModel
from diffusers.utils import load_image
import torch 

path = "sayakpaul/FLUX.1-dev-edit-v0" 
edit_transformer = FluxTransformer2DModel.from_pretrained(path, torch_dtype=torch.bfloat16)
pipeline = FluxControlPipeline.from_pretrained(
    "black-forest-labs/FLUX.1-dev", transformer=edit_transformer, torch_dtype=torch.bfloat16
).to("cuda")

url = "https://huggingface.co/datasets/sayakpaul/sample-datasets/resolve/main/flux-edit-artifacts/assets/mushroom.jpg"
image = load_image(url) # resize as needed.
print(image.size)

prompt = "turn the color of mushroom to gray"
image = pipeline(
    control_image=image,
    prompt=prompt,
    guidance_scale=30., # change this as needed.
    num_inference_steps=50, # change this as needed.
    max_sequence_length=512,
    height=image.height,
    width=image.width,
    generator=torch.manual_seed(0)
).images[0]
image.save("edited_image.png")

高級用法

使用turbo LoRA加速推理：

from diffusers import FluxControlPipeline, FluxTransformer2DModel
from diffusers.utils import load_image
from huggingface_hub import hf_hub_download
import torch

path = "sayakpaul/FLUX.1-dev-edit-v0"
edit_transformer = FluxTransformer2DModel.from_pretrained(path, torch_dtype=torch.bfloat16)
pipeline = FluxControlPipeline.from_pretrained(
    "black-forest-labs/FLUX.1-dev", transformer=edit_transformer, torch_dtype=torch.bfloat16
).to("cuda")

# load the turbo LoRA
pipeline.load_lora_weights(
    hf_hub_download("ByteDance/Hyper-SD", "Hyper-FLUX.1-dev-8steps-lora.safetensors"), adapter_name="hyper-sd"
)
pipeline.set_adapters(["hyper-sd"], adapter_weights=[0.125])


url = "https://huggingface.co/datasets/sayakpaul/sample-datasets/resolve/main/flux-edit-artifacts/assets/mushroom.jpg"
image = load_image(url) # resize as needed.
print(image.size)

prompt = "turn the color of mushroom to gray"
image = pipeline(
    control_image=image,
    prompt=prompt,
    guidance_scale=30., # change this as needed.
    num_inference_steps=8, # change this as needed.
    max_sequence_length=512,
    height=image.height,
    width=image.width,
    generator=torch.manual_seed(0)
).images[0]
image.save("edited_image.png")

推理速度對比

對比

50 steps	8 steps

guidance_scale對結果的影響

提示	組合圖 (gs: 10)	組合圖 (gs: 20)	組合圖 (gs: 30)	組合圖 (gs: 40)
Give this the look of a traditional Japanese woodblock print.
transform the setting to a winter scene
turn the color of mushroom to gray

🔧 技術細節

訓練細節

微調代碼庫位於 這裡。訓練超參數如下：

• 每個GPU的批次大小：4
• 梯度累積步數：4
• 引導比例：30
• BF16混合精度
• AdamW優化器（來自 bitsandbytes 的8位）
• 恆定學習率：5e-5
• 權重衰減：1e-6
• 訓練步數：20000

訓練使用了一個包含8個H100的節點。

我們使用了一種簡化的流機制來進行線性插值，偽代碼如下：

sigmas = torch.rand(batch_size)
timesteps = (sigmas * noise_scheduler.config.num_train_timesteps).long()
...

noisy_model_input = (1.0 - sigmas) * pixel_latents + sigmas * noise

其中，pixel_latents 是從源圖像計算得到的，noise 是從高斯分佈中採樣得到的。更多細節請 查看倉庫。

📄 許可證

請遵守 此處 所述的許可條款。

📚 詳細文檔

預期用途和限制

推理

可以按照上述代碼示例進行推理。如果硬件內存不足，還可以選擇進行量化，更多信息請參考 Diffusers文檔。

限制和偏差

由於我們不清楚Flux Control的確切訓練細節，因此模型的表現可能不盡如人意。