๐ Controlnet - v1.1 - InPaint Version
This project focuses on the InPaint Version of Controlnet v1.1, which is a conversion of the original checkpoint into diffusers
format. It can be used in combination with Stable Diffusion to control diffusion models by adding extra conditions, enabling more diverse image generation.
๐ Quick Start
Prerequisites
You need to have a Hugging Face token (HF_TOKEN
) and an API endpoint (API_ENDPOINT
).
Installation
$ pip install diffusers transformers accelerate
Usage
import base64
import requests
HF_TOKEN = 'hf_xxxxxxxxxxxxx'
API_ENDPOINT = 'https://xxxxxxxxxxx.us-east-1.aws.endpoints.huggingface.cloud'
def load_image(path):
try:
with open(path, 'rb') as file:
return file.read()
except FileNotFoundError as error:
print('Error reading image:', error)
def get_b64_image(path):
image_buffer = load_image(path)
if image_buffer:
return base64.b64encode(image_buffer).decode('utf-8')
def process_images(original_image_path, mask_image_path, result_path, prompt, width, height):
original_b64 = get_b64_image(original_image_path)
mask_b64 = get_b64_image(mask_image_path)
if not original_b64 or not mask_b64:
return
body = {
'inputs': prompt,
'image': original_b64,
'mask_image': mask_b64,
'width': width,
'height': height
}
headers = {
'Authorization': f'Bearer {HF_TOKEN}',
'Content-Type': 'application/json',
'Accept': 'image/png'
}
response = requests.post(
API_ENDPOINT,
json=body,
headers=headers
)
blob = response.content
save_image(blob, result_path)
def save_image(blob, file_path):
with open(file_path, 'wb') as file:
file.write(blob)
print('File saved successfully!')
if __name__ == '__main__':
original_image_path = 'images/original.png'
mask_image_path = 'images/mask.png'
result_path = 'images/result.png'
process_images(original_image_path, mask_image_path, result_path, 'cyberpunk mona lisa', 512, 768)
โจ Features
- Control Diffusion Models: Controlnet is a neural network structure that can control diffusion models by adding extra conditions.
- Inpaint Functionality: This specific checkpoint is designed for image inpainting, allowing you to modify or complete images.
- Compatibility: It can be used in combination with Stable Diffusion, such as runwayml/stable-diffusion-v1-5.
๐ Documentation
Model Details
Introduction
Controlnet was proposed in Adding Conditional Control to Text-to-Image Diffusion Models by Lvmin Zhang and Maneesh Agrawala. The abstract states that ControlNet is a neural network structure to control pretrained large diffusion models to support additional input conditions. It can learn task-specific conditions in an end-to-end way, and the learning is robust even with a small training dataset. Training a ControlNet is as fast as fine-tuning a diffusion model, and it can be trained on personal devices or scaled to large amounts of data with powerful computation clusters.
Example
It is recommended to use this checkpoint with Stable Diffusion v1-5 as it has been trained on it. Experimentally, it can also be used with other diffusion models such as dreamboothed stable diffusion.
import torch
import os
from diffusers.utils import load_image
from PIL import Image
import numpy as np
from diffusers import (
ControlNetModel,
StableDiffusionControlNetPipeline,
UniPCMultistepScheduler,
)
checkpoint = "lllyasviel/control_v11p_sd15_inpaint"
original_image = load_image(
"https://huggingface.co/lllyasviel/control_v11p_sd15_inpaint/resolve/main/images/original.png"
)
mask_image = load_image(
"https://huggingface.co/lllyasviel/control_v11p_sd15_inpaint/resolve/main/images/mask.png"
)
def make_inpaint_condition(image, image_mask):
image = np.array(image.convert("RGB")).astype(np.float32) / 255.0
image_mask = np.array(image_mask.convert("L"))
assert image.shape[0:1] == image_mask.shape[0:1], "image and image_mask must have the same image size"
image[image_mask < 128] = -1.0
image = np.expand_dims(image, 0).transpose(0, 3, 1, 2)
image = torch.from_numpy(image)
return image
control_image = make_inpaint_condition(original_image, mask_image)
prompt = "best quality"
negative_prompt="lowres, bad anatomy, bad hands, cropped, worst quality"
controlnet = ControlNetModel.from_pretrained(checkpoint, torch_dtype=torch.float16)
pipe = StableDiffusionControlNetPipeline.from_pretrained(
"runwayml/stable-diffusion-v1-5", controlnet=controlnet, torch_dtype=torch.float16
)
pipe.scheduler = UniPCMultistepScheduler.from_config(pipe.scheduler.config)
pipe.enable_model_cpu_offload()
generator = torch.manual_seed(2)
image = pipe(prompt, negative_prompt=negative_prompt, num_inference_steps=30,
generator=generator, image=control_image).images[0]
image.save('images/output.png')

Other released checkpoints v1-1
The authors released 14 different checkpoints, each trained with Stable Diffusion v1-5 on a different type of conditioning:
๐ง Technical Details
ControlNet is a neural network structure to control diffusion models by adding extra conditions. The original checkpoint was released in lllyasviel/ControlNet-v1-1 by Lvmin Zhang. This conversion into diffusers
format allows for easier integration with Stable Diffusion.
For more details, please also have a look at the ๐งจ Diffusers docs.
๐ License
The project is licensed under The CreativeML OpenRAIL M license, which is an Open RAIL M license.