Nexus Gen

🚀 Nexus-Gen

Nexus-Gen is a unified model that combines the language reasoning capabilities of LLMs with the image synthesis power of diffusion models, enabling comprehensive handling of image understanding, generation, and editing tasks.

🚀 Quick Start

📦 Installation

1. Install DiffSynth-Studio from source:

git clone https://github.com/modelscope/DiffSynth-Studio.git
cd DiffSynth-Studio
pip install -e .

2. Install requirements

pip install -r requirements.txt

3. Install ms-swift if you want to perform finetuning on Nexus-Gen.

pip install ms-swift -U

Prepare models

python download_models.py

💻 Usage Examples

Basic Usage

Image Understanding

python image_understanding.py

Image Generation

• Image generation with detailed prompt.

python image_generation.py

• Polish prompt and generate images with Nexus-Gen.

image_generation_with_selfpolish.py

Image Editing

python image_editing.py

Advanced Usage

Nexus-Gen is trained base on ms-swift and DiffSynth-Studio. You can find the training scripts in train/scripts/train_decoder.sh and train_llm.sh.

📚 Documentation

What is the Nexus-Gen

Nexus-Gen is a unified model that synergizes the language reasoning capabilities of LLMs with the image synthesis power of diffusion models. To align the embedding space of the LLM and diffusion model, we conduct a dual - phase alignment training process. (1) The autoregressive LLM learns to predict image embeddings conditioned on multimodal inputs, while (2) the vision decoder is trained to reconstruct high - fidelity images from these embeddings. During training the LLM, we identified a critical discrepancy between the autoregressive paradigm's training and inference phases, where error accumulation in continuous embedding space severely degrades generation quality. To avoid this issue, we introduce a prefilled autoregression strategy that prefills input sequence with position - embedded special tokens instead of continuous embeddings. Through dual - phase training, Nexus - Gen has developed the integrated capability to comprehensively address the image understanding, generation and editing tasks as follows.

More information please refer to our repo: https://github.com/modelscope/Nexus - Gen.git

📄 License

This project is under the Apache-2.0 license.