Controlnet Xl Pro

🚀 ControlNet++: All-in-one ControlNet for image generations and editing!

ControlNet++ is an all - in - one solution for image generation and editing. It offers a powerful ProMax model with 12 controls and 5 advanced editing features, enabling high - quality image creation.

🚀 Quick Start

Inference scripts and more details can be found at: https://github.com/xinsir6/ControlNetPlus/tree/main

✨ Features

ProMax Model Release

The ProMax Model has been released! It comes with 12 controls and 5 advanced editing features. Just give it a try!

Network Architecture

Advantages of the Model

• High - Resolution Image Generation: Similar to novelai, it uses bucket training, which allows generating high - resolution images of any aspect ratio.
• Large - Scale High - Quality Data: Trained with over 10000000 high - quality images, the dataset covers a wide range of scenarios.
• Good Prompt - Following Ability: Similar to DALLE.3, it uses re - captioned prompts. CogVLM is used to generate detailed descriptions, ensuring excellent prompt - following performance.
• Effective Training Tricks: During training, it employs various useful techniques, including but not limited to data augmentation, multiple loss functions, and multi - resolution training.
• Low Parameter Increase: It uses almost the same parameters as the original ControlNet, without a significant increase in network parameters or computation.
• Multiple Control Conditions: Supports 10+ control conditions, with no obvious performance drop on any single condition compared to independent training.
• Multi - Condition Generation: Supports multi - condition generation, and condition fusion is learned during training. There is no need to set hyperparameters or design complex prompts.
• Compatibility: Compatible with other open - source SDXL models, such as BluePencilXL and CounterfeitXL, as well as other Lora models.

We designed a new architecture that can support 10+ control types in text - to - image generation and produce high - resolution images visually comparable to those of Midjourney. Based on the original ControlNet architecture, we proposed two new modules: 1. Extend the original ControlNet to support different image conditions using the same network parameters. 2. Support multiple condition inputs without increasing the computational burden, which is particularly important for designers who need to edit images in detail. Different conditions share the same condition encoder, without adding extra computations or parameters. We conducted thorough experiments on SDXL and achieved excellent performance in both control ability and aesthetic score. We released this method and the model to the open - source community for everyone to use.

💻 Usage Examples

Advanced Editing Features in Promax Model

Tile Deblur

Tile Variation

Tile Super Resolution

The following examples show the transition from 1M resolution to 9M resolution:

Image Inpainting

Image Outpainting

Visual Examples

Openpose

Depth

Canny

Lineart

AnimeLineart

Mlsd

Scribble

Hed

Pidi(Softedge)

Teed

Segment

Normal

Multi - Control Visual Examples

Openpose + Canny

Openpose + Depth

Openpose + Scribble

Openpose + Normal

Openpose + Segment

📄 License

This project is licensed under the Apache - 2.0 license.

💡 Usage Tip

If you find this project useful, please give it a star on GitHub. Thank you very much!

The SDXL ProMax version has been released! Enjoy it!

I'm sorry that due to the difficulty in balancing the project's revenue and expenditure, GPU resources have been allocated to other more profitable projects. SD3 training has been stopped until I can secure enough GPU support. I will do my best to find GPUs to resume training. If this causes any inconvenience, I sincerely apologize. Thank you for your support!

Note: The promax model with a promax suffix is placed in the same huggingface model repo. Detailed instructions will be added later.