FLUX.1 Dev Q8 Fp16 Fp32 Mix 8 To 32 Bpw Gguf

🚀 Flux.1-dev Quantized Model

This project offers Flux.1-dev in experimental custom formats, aiming to explore the potential of mixed tensor formats in image generation.

✨ Features

• Mixed Tensor Formats: Mixes tensors in Q8_0, fp16, and fp32 to potentially improve quality.
• Lossless Conversion: Converted from black - forest - labs' original bf16 weights, with fp32 conversion being lossless.
• Multiple Quantized Versions: Provides different quantized versions for users to choose from.

📦 Installation

The model files can be used with the [ComfyUI - GGUF](https://github.com/city96/ComfyUI - GGUF) custom node. Place model files in ComfyUI/models/unet. Refer to the GitHub readme for further install instructions.

💻 Usage Examples

Basic Usage

There is no specific code example provided in the original README. But you can use the model with the [ComfyUI - GGUF](https://github.com/city96/ComfyUI - GGUF) custom node as described in the installation section.

📚 Documentation

Motivation

Flux's weights were published in bf16. Conversion to fp16 is slightly lossy, but fp32 is lossless. The author experimented with mixed tensor formats to see if it would improve quality.

Evaluation

The author tried comparing the outputs but couldn't say with certainty if these models are significantly better than pure Q8_0. Higher bits per weight (bpw) numbers result in slower computation:

 20 s  Q8_0
 23 s  11.024bpw - txt16.gguf
 30 s  fp16
 37 s  16.422bpw - txt32.gguf
310 s  fp32

Update 2024 - 08 - 26

Two new files are added. This time the only tensors in Q8_0 are some or all of:

double_blocks.*.img_mlp.0.weight
double_blocks.*.img_mlp.2.weight
double_blocks.*.txt_mlp.0.weight
double_blocks.*.txt_mlp.2.weight

double_blocks.*.img_mod.lin.weight
double_blocks.*.txt_mod.lin.weight
single_blocks.*.linear1.weight
single_blocks.*.linear2.weight
single_blocks.*.modulation.lin.weight

• flux1 - dev - Q8_0 - fp32 - 11.763bpw.gguf
  This version has all the above layers in Q8_0.
• flux1 - dev - Q8_0 - fp32 - 13.962bpw.gguf
  This version preserves first 2 layers of all kinds, and first 4 MLP layers in fp32.
• flux1 - dev - Q8_0 - fp32 - 16.161bpw.gguf
  This one, first 4 layers of any kind and first 8 MLP layers in fp32.

In the txt16/32 files, the author quantized only these layers to Q8_0, unless they were one - dimensional:

img_mlp.0
img_mlp.2
img_mod.lin
linear1
linear2
modulation.lin

But left all these at fp16 or fp32, respectively:

txt_mlp.0
txt_mlp.2
txt_mod.lin

The resulting bpw number is just an approximation from file size.

General Information

This is a direct GGUF conversion of [black - forest - labs/FLUX.1 - dev](https://huggingface.co/black - forest - labs/FLUX.1 - dev/tree/main). As this is a quantized model not a finetune, all the same restrictions/original license terms still apply. Please refer to [this chart](https://github.com/ggerganov/llama.cpp/blob/master/examples/perplexity/README.md#llama - 3 - 8b - scoreboard) for a basic overview of quantization types.

(Model card mostly copied from [city96/FLUX.1 - dev - gguf](https://huggingface.co/city96/FLUX.1 - dev - gguf) - which contains conventional and useful GGUF files.)

📄 License

• License Name: flux - 1 - dev - non - commercial - license
• License Link: LICENSE.md

📋 Information Table

💡 Usage Tip You're probably better off using Q8_0, but these mixed tensor format models are shared in case someone finds them useful.