FLUX.1 Dev Edit V0

✨ 主要特性

• 基于特定模型和大规模数据集训练，具备强大的图像编辑能力。
• 支持使用Flux Control框架进行微调。
• 可通过不同参数设置（如 guidance_scale）和技术（如turbo LoRA、量化）优化推理效果和性能。

💻 使用示例

基础用法

from diffusers import FluxControlPipeline, FluxTransformer2DModel
from diffusers.utils import load_image
import torch 

path = "sayakpaul/FLUX.1-dev-edit-v0" 
edit_transformer = FluxTransformer2DModel.from_pretrained(path, torch_dtype=torch.bfloat16)
pipeline = FluxControlPipeline.from_pretrained(
    "black-forest-labs/FLUX.1-dev", transformer=edit_transformer, torch_dtype=torch.bfloat16
).to("cuda")

url = "https://huggingface.co/datasets/sayakpaul/sample-datasets/resolve/main/flux-edit-artifacts/assets/mushroom.jpg"
image = load_image(url) # resize as needed.
print(image.size)

prompt = "turn the color of mushroom to gray"
image = pipeline(
    control_image=image,
    prompt=prompt,
    guidance_scale=30., # change this as needed.
    num_inference_steps=50, # change this as needed.
    max_sequence_length=512,
    height=image.height,
    width=image.width,
    generator=torch.manual_seed(0)
).images[0]
image.save("edited_image.png")

高级用法

使用turbo LoRA加速推理：

from diffusers import FluxControlPipeline, FluxTransformer2DModel
from diffusers.utils import load_image
from huggingface_hub import hf_hub_download
import torch

path = "sayakpaul/FLUX.1-dev-edit-v0"
edit_transformer = FluxTransformer2DModel.from_pretrained(path, torch_dtype=torch.bfloat16)
pipeline = FluxControlPipeline.from_pretrained(
    "black-forest-labs/FLUX.1-dev", transformer=edit_transformer, torch_dtype=torch.bfloat16
).to("cuda")

# load the turbo LoRA
pipeline.load_lora_weights(
    hf_hub_download("ByteDance/Hyper-SD", "Hyper-FLUX.1-dev-8steps-lora.safetensors"), adapter_name="hyper-sd"
)
pipeline.set_adapters(["hyper-sd"], adapter_weights=[0.125])


url = "https://huggingface.co/datasets/sayakpaul/sample-datasets/resolve/main/flux-edit-artifacts/assets/mushroom.jpg"
image = load_image(url) # resize as needed.
print(image.size)

prompt = "turn the color of mushroom to gray"
image = pipeline(
    control_image=image,
    prompt=prompt,
    guidance_scale=30., # change this as needed.
    num_inference_steps=8, # change this as needed.
    max_sequence_length=512,
    height=image.height,
    width=image.width,
    generator=torch.manual_seed(0)
).images[0]
image.save("edited_image.png")

推理速度对比

对比

50 steps	8 steps

guidance_scale对结果的影响

提示	组合图 (gs: 10)	组合图 (gs: 20)	组合图 (gs: 30)	组合图 (gs: 40)
Give this the look of a traditional Japanese woodblock print.
transform the setting to a winter scene
turn the color of mushroom to gray

🔧 技术细节

训练细节

微调代码库位于 这里。训练超参数如下：

• 每个GPU的批次大小：4
• 梯度累积步数：4
• 引导比例：30
• BF16混合精度
• AdamW优化器（来自 bitsandbytes 的8位）
• 恒定学习率：5e-5
• 权重衰减：1e-6
• 训练步数：20000

训练使用了一个包含8个H100的节点。

我们使用了一种简化的流机制来进行线性插值，伪代码如下：

sigmas = torch.rand(batch_size)
timesteps = (sigmas * noise_scheduler.config.num_train_timesteps).long()
...

noisy_model_input = (1.0 - sigmas) * pixel_latents + sigmas * noise

其中，pixel_latents 是从源图像计算得到的，noise 是从高斯分布中采样得到的。更多细节请 查看仓库。

📄 许可证

请遵守 此处 所述的许可条款。

📚 详细文档

预期用途和限制

推理

可以按照上述代码示例进行推理。如果硬件内存不足，还可以选择进行量化，更多信息请参考 Diffusers文档。

限制和偏差

由于我们不清楚Flux Control的确切训练细节，因此模型的表现可能不尽如人意。