Glm 4 9b Chat Abliterated GGUF

🚀 Llamacpp imatrix Quantizations of glm-4-9b-chat-abliterated

This project provides quantized versions of the glm-4-9b-chat-abliterated model using llama.cpp. It offers various quantization types to suit different hardware and performance requirements.

Model Information

🚀 Quick Start

Quantization Process

We used llama.cpp release b3634 for quantization. The original model can be found at https://huggingface.co/byroneverson/glm-4-9b-chat-abliterated. All quantizations were made using the imatrix option with a dataset from here.

Running the Model

You can run these quantized models in LM Studio.

💻 Usage Examples

Prompt Format

[gMASK] <sop> <|system|> 
{system_prompt} <|user|> 
{prompt} <|assistant|>

📦 Installation

Downloading a Single File

First, make sure you have huggingface-cli installed:

pip install -U "huggingface_hub[cli]"

Then, you can target the specific file you want:

huggingface-cli download bartowski/glm-4-9b-chat-abliterated-GGUF --include "glm-4-9b-chat-abliterated-Q4_K_M.gguf" --local-dir ./

Downloading Split Files

If the model is bigger than 50GB, it will have been split into multiple files. In order to download them all to a local folder, run:

huggingface-cli download bartowski/glm-4-9b-chat-abliterated-GGUF --include "glm-4-9b-chat-abliterated-Q8_0/*" --local-dir ./

You can either specify a new local-dir (e.g., glm-4-9b-chat-abliterated-Q8_0) or download them all in place (./).

📚 Documentation

File Download Table

Embed/Output Weights

Some of these quantizations (e.g., Q3_K_XL, Q4_K_L) use the standard quantization method with the embeddings and output weights quantized to Q8_0 instead of the normal default. Some users claim that this improves the quality, while others don't notice any difference. If you use these models, please comment with your findings. I'd like feedback to ensure these quantizations are actually useful.

Model Selection

A great write-up with charts showing various performances is provided by Artefact2 here.

The first step is to determine the size of the model you can run. You'll need to know how much RAM and/or VRAM you have.

• If you want your model to run as fast as possible, aim to fit the whole model on your GPU's VRAM. Choose a quantization with a file size 1-2GB smaller than your GPU's total VRAM.
• If you want the highest possible quality, add your system RAM and your GPU's VRAM together, then select a quantization with a file size 1-2GB smaller than that total.

Next, decide whether you want to use an 'I-quant' or a 'K-quant'.

• If you don't want to think too much, choose one of the K-quants (e.g., Q5_K_M).
• If you want more detailed information, check out the llama.cpp feature matrix. Generally, if you're aiming for below Q4 and using cuBLAS (Nvidia) or rocBLAS (AMD), consider the I-quants (e.g., IQ3_M). These are newer and offer better performance for their size.

Note that the I-quants can be used on CPU and Apple Metal, but they'll be slower than their K-quant equivalents. Also, the I-quants are not compatible with Vulcan (AMD). Make sure to double-check if you're using the rocBLAS build or the Vulcan build. At the time of writing, LM Studio has a preview with ROCm support, and other inference engines have specific builds for ROCm.

🔧 Technical Details

Some of these quants (Q3_K_XL, Q4_K_L etc) are the standard quantization method with the embeddings and output weights quantized to Q8_0 instead of what they would normally default to.

📄 License

This project uses the glm-4 license. You can find the full license details here.

👏 Credits

Thank you kalomaze and Dampf for assistance in creating the imatrix calibration dataset. Thank you ZeroWw for the inspiration to experiment with embed/output.

If you'd like to support my work, visit my ko-fi page here.