In Context LoRA

🚀 In-Context LoRA

In-Context LoRA fine-tunes text-to-image models to generate image sets with customizable intrinsic relationships, suitable for a wide range of tasks.

📢 [Project Page] [Github Repo] [Paper]

🚀 Quick Start

For a quick start, you can refer to the Project Page and Github Repo for detailed guidance.

✨ Features

• Task-Agnostic Framework: IC-LoRA serves as a general framework, but it requires task-specific fine-tuning for diverse applications.
• Customizable Image-Set Generation: You can fine-tune text-to-image models to generate image sets with customizable intrinsic relationships.
• Condition on Image-Set: You can also condition the generation of a set of images on another set of images, enabling a wide range of controllable generation applications.

📚 Documentation

Model Summary

In-Context LoRA fine-tunes text-to-image models (e.g., FLUX) to generate image sets with customizable intrinsic relationships, optionally conditioned on another set using SDEdit. It can be adapted to a wide range of tasks.

This model hub includes In-Context LoRA models across 10 tasks. MODEL ZOO details these models and their recommend settings. For more details on how these models are trained, please refer to our paper.

Key Idea

The core concept of IC-LoRA is to concatenate both condition and target images into a single composite image while using Natural Language to define the task. This approach enables seamless adaptation to a wide range of applications.

MODEL ZOO

Below lists 10 In-Context LoRA models and their recommend settings.

Community Creations Using IC-LoRA

We are thrilled to showcase the community's innovative projects leveraging In-Context LoRA (IC-LoRA). If you have additional recommendations or projects to share, please don't hesitate to send a Pull Request!

Project Name	Type	Supported Tasks	Sample Results
1. Comfyui_Object_Migration	ComfyUI Node & Workflow & LoRA Model	Clothing Migration, Cartoon Clothing to Realism, and More
2. Flux Simple Try On - In Context Lora	LoRA Model & ComfyUI Workflow	Virtual Try-on
3. Flux In Context - visual identity Lora in Comfy	ComfyUI Workflow	Visual Identity Transfer
4. Workflows Flux In Context Lora For Product Design	ComfyUI Workflow	Product Design, Role Play, and More
5. Flux Product Design - In Context Lora	LoRA Model & ComfyUI Workflow	Product Design
6. In Context lora + Character story generator + flux+ shichen	ComfyUI Workflow	Character Movie Story Generator
7. In- Context-Lora｜Cute 4koma 可爱四格漫画	LoRA Model & ComfyUI Workflow	Comic Strip Generation
8. Creative Effects & Design LoRA Pack (In-Context LORA)	LoRA Model & ComfyUI Workflow	Movie-Shot Generation and More

🔧 Technical Details

For more detailed information on how these models are trained, please refer to our paper.

📄 License

This model hub uses FLUX as the base model. Users must comply with FLUX's license when using this code. Please refer to FLUX's License for more details.

🔥 Latest News

• [2024-11-16] 🌟 The community continues to innovate with IC-LoRA! Exciting projects include models, ComfyUI nodes and workflows for Virtual Try-on, Product Design, Object Mitigation, Role Play, and more. Explore their creations in Community Creations Using IC-LoRA. Huge thanks to all contributors for their incredible efforts!

Download model

Weights for these models are available in Safetensors format.

Download them in the Files & versions tab.

Citation

If you find this work useful in your research, please consider citing:

@article{lhhuang2024iclora,
  title={In-Context LoRA for Diffusion Transformers},
  author={Huang, Lianghua and Wang, Wei and Wu, Zhi-Fan and Shi, Yupeng and Dou, Huanzhang and Liang, Chen and Feng, Yutong and Liu, Yu and Zhou, Jingren},
  journal={arXiv preprint arxiv:2410.23775},
  year={2024}
}

@article{lhhuang2024iclora,
  title={Group Diffusion Transformers are Unsupervised Multitask Learners},
  author={Huang, Lianghua and Wang, Wei and Wu, Zhi-Fan and Dou, Huanzhang and Shi, Yupeng and Feng, Yutong and Liang, Chen and Liu, Yu and Zhou, Jingren},
  journal={arXiv preprint arxiv:2410.15027},
  year={2024}
}