Flux.1 Dev Controlnet Surface Normals

🚀 Flux.1-dev: Surface Normals ControlNet

This is a Surface Normals map ControlNet for Flux.1-dev, developed by the Jasper research team.

🚀 Quick Start

This model can be used directly with the diffusers library.

💻 Usage Examples

Basic Usage

import torch
from diffusers.utils import load_image
from diffusers import FluxControlNetModel
from diffusers.pipelines import FluxControlNetPipeline

# Load pipeline
controlnet = FluxControlNetModel.from_pretrained(
  "jasperai/Flux.1-dev-Controlnet-Surface-Normals",
  torch_dtype=torch.bfloat16
)
pipe = FluxControlNetPipeline.from_pretrained(
  "black-forest-labs/FLUX.1-dev",
  controlnet=controlnet,
  torch_dtype=torch.bfloat16
)
pipe.to("cuda")

# Load a control image
control_image = load_image(
  "https://huggingface.co/jasperai/Flux.1-dev-Controlnet-Surface-Normals/resolve/main/examples/surface.jpg"
)

prompt = "a man showing stop sign in front of window"

image = pipe(
    prompt, 
    control_image=control_image,
    controlnet_conditioning_scale=0.6,
    num_inference_steps=28, 
    guidance_scale=3.5,
    height=control_image.size[1],
    width=control_image.size[0]
).images[0]
image

Advanced Usage

from controlnet_aux import NormalBaeDetector
from diffusers.utils import load_image 

normal_bae = NormalBaeDetector.from_pretrained("lllyasviel/Annotators")

normal_bae.to("cuda")

# Load an image
im = load_image(
  "https://huggingface.co/jasperai/Flux.1-dev-Controlnet-Surface-Normals/resolve/main/examples/output.jpg"
)

surface = normal_bae(im)

💡 Usage Tip

You can compute the conditioning map using the NormalBaeDetector from the controlnet_aux library.

🔧 Technical Details

This model was trained with surface normals maps computed with Clipdrop's surface normals estimator model as well as an open - source surface normals estimation model such as Boundary Aware Encoder (BAE).

📄 License

This model falls under the Flux.1-dev model licence.