Svdq Int4 Flux.1 Schnell

🚀 SVDQuant: 4-bit Quantization for Text-to-Image Models

SVDQuant is a post - training quantization technique designed for 4 - bit weights and activations, which effectively maintains visual fidelity. It offers significant memory reduction and speedup in text - to - image generation tasks.

🚀 Quick Start

To start using the svdq - int4 - flux.1 - schnell model, you need to set up the environment as described in [mit - han - lab/nunchaku](https://github.com/mit - han - lab/nunchaku).

Diffusers

import torch
from diffusers import FluxPipeline

from nunchaku.models.transformer_flux import NunchakuFluxTransformer2dModel

transformer = NunchakuFluxTransformer2dModel.from_pretrained("mit - han - lab/svdq - int4 - flux.1 - schnell")
pipeline = FluxPipeline.from_pretrained(
    "black - forest - labs/FLUX.1 - schnell", transformer=transformer, torch_dtype=torch.bfloat16
).to("cuda")
image = pipeline(
    "A cat holding a sign that says hello world", width=1024, height=1024, num_inference_steps=4, guidance_scale=0
).images[0]
image.save("flux.1 - schnell - int4.png")

Comfy UI

 For usage details, please refer to comfyui/README.md.

✨ Features

• Memory Reduction: On 12B FLUX.1 - dev, SVDQuant achieves 3.6× memory reduction compared to the BF16 model.
• Speedup: By eliminating CPU offloading, it offers 8.7× speedup over the 16 - bit model on a 16GB laptop 4090 GPU, 3× faster than the NF4 W4A16 baseline.
• Visual Quality: On PixArt - ∑, it demonstrates significantly superior visual quality over other W4A4 or even W4A8 baselines.

📦 Installation

Please follow the instructions in [mit - han - lab/nunchaku](https://github.com/mit - han - lab/nunchaku) to set up the environment.

📚 Documentation

Method

Quantization Method -- SVDQuant

 Overview of SVDQuant. Stage1: Originally, both the activation X and weights W contain outliers, making 4 - bit quantization challenging. Stage 2: We migrate the outliers from activations to weights, resulting in the updated activation and weight. While the activation becomes easier to quantize, the weight now becomes more difficult. Stage 3: SVDQuant further decomposes the weight into a low - rank component and a residual with SVD. Thus, the quantization difficulty is alleviated by the low - rank branch, which runs at 16 - bit precision.

Nunchaku Engine Design

 (a) Naïvely running low - rank branch with rank 32 will introduce 57% latency overhead due to extra read of 16 - bit inputs in Down Projection and extra write of 16 - bit outputs in Up Projection. Nunchaku optimizes this overhead with kernel fusion. (b) Down Projection and Quantize kernels use the same input, while Up Projection and 4 - Bit Compute kernels share the same output. To reduce data movement overhead, we fuse the first two and the latter two kernels together.

Model Description

🔧 Technical Details

The SVDQuant quantization method and Nunchaku engine design are the key technical components. SVDQuant addresses the challenge of 4 - bit quantization by migrating outliers and decomposing weights. The Nunchaku engine optimizes the low - rank branch to reduce latency overhead.

📄 License

This project is licensed under the Apache - 2.0 license.

Citation

If you find this model useful or relevant to your research, please cite

@inproceedings{
  li2024svdquant,
  title={SVDQuant: Absorbing Outliers by Low - Rank Components for 4 - Bit Diffusion Models},
  author={Li*, Muyang and Lin*, Yujun and Zhang*, Zhekai and Cai, Tianle and Li, Xiuyu and Guo, Junxian and Xie, Enze and Meng, Chenlin and Zhu, Jun - Yan and Han, Song},
  booktitle={The Thirteenth International Conference on Learning Representations},
  year={2025}
}

⚠️ Important Note

• The model is only runnable on NVIDIA GPUs with architectures sm_86 (Ampere: RTX 3090, A6000), sm_89 (Ada: RTX 4090), and sm_80 (A100). See this [issue](https://github.com/mit - han - lab/nunchaku/issues/1) for more details.
• You may observe some slight differences from the BF16 models in detail.