72B Qwen2.5 Kunou V1 GGUF

🚀 Llamacpp imatrix Quantizations of 72B-Qwen2.5-Kunou-v1

This project provides Llama.cpp imatrix quantizations of the 72B-Qwen2.5-Kunou-v1 model. It enables efficient deployment and usage of the model in various environments, especially in LM Studio.

🚀 Quick Start

Quantization Process

We use llama.cpp release b4273 for quantization. The original model can be found at Sao10K/72B-Qwen2.5-Kunou-v1. All quantizations are made using the imatrix option with a dataset from here.

Running the Model

You can run the quantized models in LM Studio.

Prompt Format

<|im_start|>system
{system_prompt}<|im_end|>
<|im_start|>user
{prompt}<|im_end|>
<|im_start|>assistant

✨ Features

Multiple Quantization Types

We offer a wide range of quantization types, each with different file sizes and quality levels. You can choose the one that best suits your needs based on your available resources and performance requirements.

Embed/Output Weights

Some of the quantizations (e.g., Q3_K_XL, Q4_K_L) use the standard quantization method with the embeddings and output weights quantized to Q8_0 instead of the default values.

Online Repacking for ARM

Thanks to recent efforts, you can use Q4_0 or IQ4_NL for better performance on ARM devices if your llama.cpp has been compiled accordingly.

📦 Installation

Downloading using huggingface-cli

pip install -U "huggingface_hub[cli]"

huggingface-cli download bartowski/72B-Qwen2.5-Kunou-v1-GGUF --include "72B-Qwen2.5-Kunou-v1-Q4_K_M.gguf" --local-dir ./

huggingface-cli download bartowski/72B-Qwen2.5-Kunou-v1-GGUF --include "72B-Qwen2.5-Kunou-v1-Q8_0/*" --local-dir ./

💻 Usage Examples

Downloading a Single File

huggingface-cli download bartowski/72B-Qwen2.5-Kunou-v1-GGUF --include "72B-Qwen2.5-Kunou-v1-Q4_K_M.gguf" --local-dir ./

Downloading Split Files

huggingface-cli download bartowski/72B-Qwen2.5-Kunou-v1-GGUF --include "72B-Qwen2.5-Kunou-v1-Q8_0/*" --local-dir ./

📚 Documentation

File Download Options

You can download individual files or split files depending on the model size. Refer to the "Downloading using huggingface-cli" section for detailed instructions.

Q4_0_X_X Information

New: If your llama.cpp has been compiled for your ARM device, you can use Q4_0 for online repacking of weights. Similarly, IQ4_NL can provide slightly better performance on ARM thanks to recent improvements.

Choosing the Right File

A great write-up with charts showing various performances is provided by Artefact2 here. First, determine how much RAM and/or VRAM you have to figure out the appropriate model size. If you want the model to run as fast as possible, choose a quant with a file size 1-2GB smaller than your GPU's total VRAM.

🔧 Technical Details

Quantization Types and Performance

The following table provides an overview of the available quantization types, their file sizes, split status, and descriptions:

Q4_0_X_X Performance on ARM and AVX

The Q4_0_X_X quantizations are optimized for ARM and certain AVX2/AVX512 CPUs. They are not suitable for Metal (Apple) or GPU (nvidia/AMD/intel) offloading. Check out the original pull request for Q4_0_4_4 speed comparisons.

Benchmarks on an AVX2 System (EPYC7702)

Q4_0_8_8 offers a significant improvement in prompt processing and a small improvement in text generation.

📄 License

This project uses the Qwen license.