BRIA 2.3 ControlNet Inpainting

🚀 BRIA 2.3 ControlNet Inpainting Fast

BRIA 2.3 ControlNet Inpainting Fast is trained on a large - scale commercial - grade dataset, ensuring high - quality results and full legal liability for commercial use. It can fill masked regions in images based on text prompts.

🚀 Quick Start

BRIA 2.3 ControlNet Inpainting Fast is a powerful model for text - to - image inpainting. It offers high - quality results and legal protection for commercial applications.

Join our Discord community for more information, tutorials, tools, and to connect with other users!

✨ Features

New Features

BRIA 2.3 ControlNet Inpainting can be applied on top of BRIA 2.3 Text - to - Image and therefore enable to use Fast - LORA. This results in an extremely fast inpainting model, requiring only 1.6s using A10 GPU.

General Features

• High - Quality Results: Trained on the largest multi - source commercial - grade licensed dataset, it guarantees the best quality.
• Commercial Safety: The model provides full legal liability coverage for copyright and privacy infringement and harmful content mitigation.
• Versatile Application: It can be used for object removal, replacement, addition, and modification within an image, as well as image expansion.

📦 Installation

The model weights from BRIA AI can be obtained with the purchase of a commercial license. Fill in the form below and we'll reach out to you.

Request a Commercial License

Fill in this form to request a commercial license for the model.

💻 Usage Examples

Basic Usage

Download

from huggingface_hub import hf_hub_download
import os

try:
    local_dir = os.path.dirname(__file__)
except:
    local_dir = '.'
    

hf_hub_download(repo_id="briaai/BRIA-2.3-ControlNet-Inpainting", filename='controlnet.py', local_dir=local_dir)
hf_hub_download(repo_id="briaai/BRIA-2.3-ControlNet-Inpainting", filename='config.json', local_dir=local_dir)
hf_hub_download(repo_id="briaai/BRIA-2.3-ControlNet-Inpainting", filename='image_processor.py', local_dir=local_dir)
hf_hub_download(repo_id="briaai/BRIA-2.3-ControlNet-Inpainting", filename='pipeline_controlnet_sd_xl.py', local_dir=local_dir)

Run

from diffusers import (
    AutoencoderKL,
    LCMScheduler,
)
from pipeline_controlnet_sd_xl import StableDiffusionXLControlNetPipeline
from controlnet import ControlNetModel
import torch
import numpy as np
from PIL import Image
import requests
import PIL
from io import BytesIO
from torchvision import transforms
import os 


def resize_image_to_retain_ratio(image):
    pixel_number = 1024*1024
    granularity_val = 8
    ratio = image.size[0] / image.size[1]
    width = int((pixel_number * ratio) ** 0.5)
    width = width - (width % granularity_val)
    height = int(pixel_number / width)
    height = height - (height % granularity_val)

    image = image.resize((width, height))
    return image


def download_image(url):
    response = requests.get(url)
    return PIL.Image.open(BytesIO(response.content)).convert("RGB")


def get_masked_image(image, image_mask, width, height):
    image_mask = image_mask # inpaint area is white
    image_mask = image_mask.resize((width, height)) # object to remove is white (1)
    image_mask_pil = image_mask
    image = np.array(image.convert("RGB")).astype(np.float32) / 255.0
    image_mask = np.array(image_mask_pil.convert("L")).astype(np.float32) / 255.0
    assert image.shape[0:1] == image_mask.shape[0:1], "image and image_mask must have the same image size"
    masked_image_to_present = image.copy()
    masked_image_to_present[image_mask > 0.5] = (0.5,0.5,0.5)  # set as masked pixel
    image[image_mask > 0.5] = 0.5  # set as masked pixel - s.t. will be grey 
    image = Image.fromarray((image * 255.0).astype(np.uint8))
    masked_image_to_present = Image.fromarray((masked_image_to_present * 255.0).astype(np.uint8))
    return image, image_mask_pil, masked_image_to_present


image_transforms = transforms.Compose(
    [
        transforms.ToTensor(),
    ]
)

default_negative_prompt = "Logo,Watermark,Text,Ugly,Morbid,Extra fingers,Poorly drawn hands,Mutation,Blurry,Extra limbs,Gross proportions,Missing arms,Mutated hands,Long neck,Duplicate,Mutilated,Mutilated hands,Poorly drawn face,Deformed,Bad anatomy,Cloned face,Malformed limbs,Missing legs,Too many fingers"

img_url = "https://raw.githubusercontent.com/CompVis/latent-diffusion/main/data/inpainting_examples/overture-creations-5sI6fQgYIuo.png"
mask_url = "https://raw.githubusercontent.com/CompVis/latent-diffusion/main/data/inpainting_examples/overture-creations-5sI6fQgYIuo_mask.png"

init_image = download_image(img_url).resize((1024, 1024))
mask_image = download_image(mask_url).resize((1024, 1024))


init_image = resize_image_to_retain_ratio(init_image)
width, height = init_image.size

mask_image = mask_image.convert("L").resize(init_image.size)

width, height = init_image.size

# Load, init model    
controlnet = ControlNetModel().from_pretrained("briaai/BRIA-2.3-ControlNet-Inpainting", torch_dtype=torch.float16)
vae = AutoencoderKL.from_pretrained("madebyollin/sdxl-vae-fp16-fix", torch_dtype=torch.float16)
pipe = StableDiffusionXLControlNetPipeline.from_pretrained("briaai/BRIA-2.3", controlnet=controlnet.to(dtype=torch.float16), torch_dtype=torch.float16, vae=vae) #force_zeros_for_empty_prompt=False, # vae=vae)

pipe.scheduler = LCMScheduler.from_config(pipe.scheduler.config)
pipe.load_lora_weights("briaai/BRIA-2.3-FAST-LORA")
pipe.fuse_lora()
pipe = pipe.to(device="cuda")

# pipe.enable_xformers_memory_efficient_attention()

generator = torch.Generator(device="cuda").manual_seed(123456)

vae = pipe.vae


masked_image, image_mask, masked_image_to_present = get_masked_image(init_image, mask_image, width, height)

masked_image_tensor = image_transforms(masked_image)
masked_image_tensor = (masked_image_tensor - 0.5) / 0.5


masked_image_tensor = masked_image_tensor.unsqueeze(0).to(device="cuda")
control_latents = vae.encode(  
        masked_image_tensor[:, :3, :, :].to(vae.dtype)
    ).latent_dist.sample()   
control_latents = control_latents * vae.config.scaling_factor 


image_mask = np.array(image_mask)[:,:]
mask_tensor = torch.tensor(image_mask, dtype=torch.float32)[None, ...]
# binarize the mask
mask_tensor = torch.where(mask_tensor > 128.0, 255.0, 0)       

mask_tensor = mask_tensor / 255.0

mask_tensor = mask_tensor.to(device="cuda")
mask_resized = torch.nn.functional.interpolate(mask_tensor[None, ...], size=(control_latents.shape[2], control_latents.shape[3]), mode='nearest')

masked_image = torch.cat([control_latents, mask_resized], dim=1)

prompt = ""

gen_img = pipe(negative_prompt=default_negative_prompt, prompt=prompt, 
            controlnet_conditioning_scale=1.0, 
            num_inference_steps=12, 
            height=height, width=width, 
            image = masked_image, # control image
            init_image = init_image,     
            mask_image = mask_tensor,
            guidance_scale = 1.2,
            generator=generator).images[0]


display(gen_img)

📚 Documentation

Model Description

Get Access to the source code and pre - trained model

Interested in BRIA 2.3 inpainting? Our Model is available for purchase.

Purchasing access to BRIA 2.3 inpainting ensures royalty management and full liability for commercial use.

Are you a startup or a student? We encourage you to apply for our specialized Academia and [Startup Programs](https://pages.bria.ai/the - visual - generative - ai - platform - for - builders - startups - plan?_gl=1cqrl81_gaMTIxMDI2NzI5OC4xNjk5NTQ3MDAz_ga_WRN60H46X4*MTcwOTM5OTMzNC4yNzguMC4xNzA5Mzk5MzM0LjYwLjAuMA..) to gain access. These programs are designed to support emerging businesses and academic pursuits with our cutting - edge technology.

Contact us today to unlock the potential of BRIA 2.3 inpainting!

By submitting the form above, you agree to BRIA’s [Privacy policy](https://bria.ai/privacy - policy/) and [Terms & conditions](https://bria.ai/terms - and - conditions/).

📄 License

The model is under the [bria - 2.3 inpainting Licensing terms & conditions](https://bria.ai/bria - huggingface - model - license - agreement/).