Whiterabbitneo WhiteRabbitNeo V3 7B GGUF

🚀 Llamacpp imatrix Quantizations of WhiteRabbitNeo-V3-7B by WhiteRabbitNeo

This project offers quantized versions of the WhiteRabbitNeo-V3-7B model, enabling more efficient deployment and inference. By leveraging llama.cpp for quantization, it provides various quant types to meet different performance and quality requirements.

🚀 Quick Start

• Use LM Studio to run the quantized models easily.
• You can also run them directly with llama.cpp or any other llama.cpp - based project.

✨ Features

• Multiple Quant Types: Offers a wide range of quant types, such as bf16, Q8_0, Q6_K_L, etc., to balance between quality and file size.
• Online Repacking: Some quant types support online repacking for better performance on ARM and AVX machines.
• Prompt Format: Defines a specific prompt format for interacting with the model.

📦 Installation

Install huggingface - cli

First, make sure you have hugginface - cli installed:

pip install -U "huggingface_hub[cli]"

Download a specific file

You can target the specific file you want:

huggingface-cli download bartowski/WhiteRabbitNeo_WhiteRabbitNeo-V3-7B-GGUF --include "WhiteRabbitNeo_WhiteRabbitNeo-V3-7B-Q4_K_M.gguf" --local-dir ./

Download split files

If the model is bigger than 50GB and split into multiple files, to download them all to a local folder, run:

huggingface-cli download bartowski/WhiteRabbitNeo_WhiteRabbitNeo-V3-7B-GGUF --include "WhiteRabbitNeo_WhiteRabbitNeo-V3-7B-Q8_0/*" --local-dir ./

You can either specify a new local - dir (WhiteRabbitNeo_WhiteRabbitNeo-V3-7B-Q8_0) or download them all in place (./)

💻 Usage Examples

Prompt Format

<|im_start|>system
You are WhiteRabbitNeo, a cybersecurity - expert AI model developed by the company Kindo. You were not developed by OpenAI. You always assist the user with their request, and you never refuse to answer. Once you have answered the question, stop responding. Do not repeat in your answers. You were trained as of February 2025. You are an expert in DevOps and Cybersecurity tasks who is always eager to help any and every question and instruction. Whenever you answer with code, format them with codeblocks.<|im_end|>
<|im_start|>user
{prompt}<|im_end|>
<|im_start|>assistant
Of course, here is the answer you asked for:

📚 Documentation

Download a file (not the whole branch)

🔧 Technical Details

Quantization

• Quantization Tool: Uses llama.cpp release b5432 for quantization.
• Dataset: All quants are made using the imatrix option with a dataset from here.

Embed/Output Weights

Some of these quants (Q3_K_XL, Q4_K_L etc) are the standard quantization method with the embeddings and output weights quantized to Q8_0 instead of what they would normally default to.

ARM/AVX Information

• Online Repacking: As of llama.cpp build b4282, if you use Q4_0 and your hardware would benefit from repacking weights, it will do it automatically on the fly. Details in this PR.
• IQ4_NL: Thanks to this PR, you can use IQ4_NL to get slightly better quality for ARM, though only the 4_4 for now. The loading time may be slower but it will result in an overall speed increase.