Gemma 2 9b It Abliterated GGUF

🚀 Llamacpp imatrix Quantizations of gemma-2-9b-it-abliterated

This project provides Llama.cpp imatrix quantizations of the gemma-2-9b-it-abliterated model. It offers various quantization types to meet different performance and quality requirements, and provides guidance on downloading and using these quantized models.

🚀 Quick Start

Prerequisites

• Ensure you have huggingface-cli installed. You can install it using the following command:

pip install -U "huggingface_hub[cli]"

Downloading a Specific File

To download a specific quantized model file, use the following command. For example, to download gemma-2-9b-it-abliterated-Q4_K_M.gguf:

huggingface-cli download bartowski/gemma-2-9b-it-abliterated-GGUF --include "gemma-2-9b-it-abliterated-Q4_K_M.gguf" --local-dir ./

Downloading Split Files

If the model is split into multiple files (models larger than 50GB), you can download all the files to a local folder using the following command:

huggingface-cli download bartowski/gemma-2-9b-it-abliterated-GGUF --include "gemma-2-9b-it-abliterated-Q8_0/*" --local-dir ./

Running the Model

You can run these quantized models in LM Studio.

✨ Features

• Multiple Quantization Types: Offers a wide range of quantization types, including f32, Q8_0, Q6_K_L, Q5_K_L, etc., to balance between model quality and file size.
• Optimized for Different Hardware: Some quantization types are optimized for ARM chips, providing significant speed improvements.
• Embed/Output Weights Option: Some quantizations use Q8_0 for embed and output weights, which may improve model quality.

📦 Installation

Installing huggingface-cli

pip install -U "huggingface_hub[cli]"

Downloading Specific Files

huggingface-cli download bartowski/gemma-2-9b-it-abliterated-GGUF --include "gemma-2-9b-it-abliterated-Q4_K_M.gguf" --local-dir ./

Downloading Split Files

huggingface-cli download bartowski/gemma-2-9b-it-abliterated-GGUF --include "gemma-2-9b-it-abliterated-Q8_0/*" --local-dir ./

💻 Usage Examples

Prompt Format

<bos><start_of_turn>system
{system_prompt}<end_of_turn>
<start_of_turn>user
{prompt}<end_of_turn>
<start_of_turn>model
<end_of_turn>
<start_of_turn>model

📚 Documentation

Model Information

Downloadable Files

Embed/Output Weights

Some of these quants (Q3_K_XL, Q4_K_L etc) use the standard quantization method with the embeddings and output weights quantized to Q8_0 instead of the default. Some users claim this improves the quality, while others notice no difference. Please share your findings if you use these models.

Q4_0_X_X

These quantizations are optimized for ARM chips, not for Metal (Apple) offloading. They can provide a substantial speedup on ARM chips. Check the AArch64 SoC features to find the best option for your ARM chip.

File Selection

A detailed analysis with performance charts is available here. Consider your available RAM and VRAM when choosing a model. For maximum speed, select a quantized model with a file size 1-2GB smaller than your GPU's VRAM. For maximum quality, combine your system RAM and GPU's VRAM and choose a model 1-2GB smaller than the total. You can also choose between 'I-quants' and 'K-quants' based on your specific requirements.

🔧 Technical Details

Quantization Method

The quantizations are performed using llama.cpp release b3878. All quants are made using the imatrix option with a dataset from here.

ARM Optimization

The Q4_0_X_X quants are optimized for ARM chips and can provide significant speed improvements. Check the original pull request for speed comparisons.

📄 License

The project uses the gemma license.

Credits

• Thank you kalomaze and Dampf for assistance in creating the imatrix calibration dataset.
• Thank you ZeroWw for the inspiration to experiment with embed/output.

If you want to support the developer's work, visit the ko-fi page.