Gemma 2 2b It Abliterated GGUF

🚀 Llamacpp imatrix Quantizations of gemma-2-2b-it-abliterated

This project provides Llama.cpp imatrix quantizations of the gemma-2-2b-it-abliterated model. It offers various quantization types to meet different performance and quality requirements, allowing users to run the model efficiently on different hardware configurations.

🚀 Quick Start

Prerequisites

Make sure you have the huggingface-cli installed. You can install it using the following command:

pip install -U "huggingface_hub[cli]"

Downloading a Specific File

To download a specific file, use the following command. For example, to download the gemma-2-2b-it-abliterated-Q4_K_M.gguf file:

huggingface-cli download bartowski/gemma-2-2b-it-abliterated-GGUF --include "gemma-2-2b-it-abliterated-Q4_K_M.gguf" --local-dir ./

Downloading Split Files

If the model is split into multiple files (models larger than 50GB), you can download all the files in a specific split to a local folder using the following command:

huggingface-cli download bartowski/gemma-2-2b-it-abliterated-GGUF --include "gemma-2-2b-it-abliterated-Q8_0/*" --local-dir ./

You can either specify a new local directory or download them all in the current directory (./).

Running the Model

You can run the quantized models in LM Studio.

✨ Features

• Multiple Quantization Types: Offers a wide range of quantization types, including f32, Q8_0, Q6_K_L, etc., to balance between model quality and file size.
• Embed/Output Weights Optimization: Some quantizations use Q8_0 for embeddings and output weights, which may improve model quality.
• Easy Download: Supports downloading specific files or split files using the huggingface-cli.

📦 Installation

The installation mainly involves installing the huggingface-cli and downloading the desired quantized model files. Refer to the "Quick Start" section for detailed steps.

💻 Usage Examples

Prompt Format

The following is the prompt format for this model. Note that this model does not support a System prompt.

<bos><start_of_turn>user
{prompt}<end_of_turn>
<start_of_turn>model
<end_of_turn>
<start_of_turn>model

📚 Documentation

Model Information

Downloadable Files

Embed/Output Weights

Some of these quants (Q3_K_XL, Q4_K_L etc) use the standard quantization method with the embeddings and output weights quantized to Q8_0 instead of the normal default. Some users claim that this improves the quality, while others don't notice any difference. If you use these models, please comment with your findings.

🔧 Technical Details

Quantization Process

The quantization is performed using llama.cpp release b3496. All quants are made using the imatrix option with the dataset from here.

File Selection

A great write-up with charts showing various performances is provided by Artefact2 here. To choose the appropriate file, first, determine how much RAM and/or VRAM you have. If you want the model to run as fast as possible, try to fit the whole model on your GPU's VRAM. If you want the maximum quality, add your system RAM and GPU's VRAM together and choose a quant with a file size 1 - 2GB smaller than the total.

📄 License

The model is licensed under the gemma license.

Credits

• Thank you kalomaze and Dampf for assistance in creating the imatrix calibration dataset.
• Thank you ZeroWw for the inspiration to experiment with embed/output.

💡 Usage Tip

If you want to support the developer's work, you can visit the ko-fi page here: https://ko-fi.com/bartowski