Controlnet Union Sdxl 1.0 Promax

🚀 ControlNet++: All-in-one ControlNet for image generations and editing!

ControlNet++ is an all - in - one solution for image generation and editing. It offers a ProMax model with 12 controls and 5 advanced editing features, enabling high - quality image outputs comparable to Midjourney.

🚀 Quick Start

Inference scripts and more details can be found at: https://github.com/xinsir6/ControlNetPlus/tree/main

✨ Features

ProMax Model Release

The ProMax Model has been released! It comes with 12 controls and 5 advanced editing features. Just give it a try!

Network Architecture

Advantages of the Model

• High - Resolution Image Generation: Utilizes bucket training similar to NovelAI, capable of generating high - resolution images of any aspect ratio.
• High - Quality Data: Trained on a large amount of high - quality data (over 10,000,000 images), covering a wide variety of scenarios.
• Prompt Following: Employs re - captioned prompts like DALLE.3, using CogVLM to generate detailed descriptions, resulting in excellent prompt - following ability.
• Training Tricks: Applies numerous useful tricks during training, including but not limited to data augmentation, multiple loss functions, and multi - resolution training.
• Parameter Efficiency: Has almost the same number of parameters as the original ControlNet, with no significant increase in network parameters or computation.
• Multiple Control Conditions: Supports 10+ control conditions, with no obvious performance drop on any single condition compared to independent training.
• Multi - Condition Generation: Supports multi - condition generation, where condition fusion is learned during training, eliminating the need to set hyperparameters or design complex prompts.
• Compatibility: Compatible with other open - source SDXL models, such as BluePencilXL and CounterfeitXL, as well as other Lora models.

New Architecture Design

We've designed a new architecture that can support 10+ control types in text - to - image generation and produce high - resolution images visually comparable to Midjourney. Based on the original ControlNet architecture, we've introduced two new modules:

1. Extend the original ControlNet to support different image conditions using the same network parameters.
2. Enable multiple - condition input without increasing computational offload, which is crucial for designers who need detailed image editing. Different conditions share the same condition encoder, without adding extra computations or parameters.

We've conducted thorough experiments on SDXL and achieved superior performance in both control ability and aesthetic score. We're releasing the method and the model to the open - source community for everyone to enjoy.

💻 Usage Examples

Advanced Editing Features in ProMax Model

Tile Deblur

Tile Variation

Tile Super Resolution

The following examples show the transformation from 1M resolution to 9M resolution:

Image Inpainting

Image Outpainting

Visual Examples

Openpose

Depth

Canny

Lineart

AnimeLineart

Mlsd

Scribble

Hed

Pidi(Softedge)

Teed

Segment

Normal

Multi - Control Visual Examples

Openpose + Canny

Openpose + Depth

Openpose + Scribble

Openpose + Normal

Openpose + Segment

📄 License

This project is licensed under the Apache - 2.0 license.

💡 Usage Tip

If you find this project useful, please give it a star. Thanks a lot!

~500+ stars, release the ProMax version with tile and inpainting!!!~
The SDXL ProMax version has been released!!! Enjoy it!!!
1000+ stars, release the ControlNet++ model for SD3!!!
3000+ stars, release the ControlNet++ ProMax model for SD3!!!

Note: We've placed the promax model with a promax suffix in the same huggingface model repo. Detailed instructions will be added later.