Controlnet Seg Room

🚀 Interior Design ControlNet Model

This project is a custom ControlNet model trained for high - quality interior design image generation, leveraging the power of segmentation maps and prompts.

🚀 Quick Start

The model is available in a community pipeline that supports image - to - image and inpainting. You can test its capabilities on the following streamlit demo: ControlNet Interior Design Demo

✨ Features

• Fine - grained Control: Allows end - users to precisely control which objects to place in the room by using segmentation maps.
• Custom Dataset: Trained on a custom dataset built from LAION5B, containing 130k images of 15 types of rooms in nearly 30 design styles.
• Multi - framework Compatibility: Converted from a JAX checkpoint to a PyTorch checkpoint for easy integration with the diffusers library.

📦 Installation

No specific installation steps are provided in the original document, so this section is skipped.

💻 Usage Examples

No code examples are provided in the original document, so this section is skipped.

📚 Documentation

📄 License

The model is released under the OpenRAIL license.

🔍 About the Dataset

To optimize this demo, our team spent a significant amount of time training a custom model. We constructed our custom dataset using the LAION5B dataset, which includes 130k images of 15 types of rooms in almost 30 design styles. After collecting these images, we added metadata such as captions (generated by the BLIP captioning model) and segmentation maps (produced by the HuggingFace UperNetForSemanticSegmentation model).

🤖 About the Model

This dataset was used to train the ControlNet model to generate high - quality interior design images. The model uses segmentation maps and prompts as conditioning information. By training on segmentation maps, users can have fine - grained control over the objects in the room. The training began with the lllyasviel/control_v11p_sd15_seg checkpoint, a well - trained ControlNet model conditioned on segmentation maps. This checkpoint was fine - tuned on a TPUv4 using the JAX framework and then converted into a PyTorch checkpoint for seamless integration with the diffusers library.

🎉 About the Demo

Our team developed a Streamlit demo where you can test the model's capabilities. The resulting model is integrated into a community pipeline that supports image - to - image and inpainting, enabling users to retain elements of their room and modify specific parts of the image.

🔧 Technical Details

No detailed technical implementation details are provided in the original document, so this section is skipped.

🙏 Thanks

Big thanks to Google for lending us TPUv4s to train this model. Big thanks to the Huggingface and Diffusers team for organizing the JAX Diffusers sprint, providing support, and creating the JAX training scripts. Big thanks to StabilityAI for open - sourcing the Stable Diffusion model, which has had a profound impact on the community!