Phi 3.5 Mini Instruct Uncensored GGUF

🚀 Llamacpp imatrix Quantizations of Phi-3.5-mini-instruct_Uncensored

This project provides llama.cpp imatrix quantizations of the Phi-3.5-mini-instruct_Uncensored model. It offers various quantization types to balance between model quality and resource usage.

🚀 Quick Start

Prerequisites

• Ensure you have huggingface-cli installed. You can install it using the following command:

pip install -U "huggingface_hub[cli]"

Download a Model

You can download a specific model file using the huggingface-cli. For example, to download the Phi-3.5-mini-instruct_Uncensored-Q4_K_M.gguf file:

huggingface-cli download bartowski/Phi-3.5-mini-instruct_Uncensored-GGUF --include "Phi-3.5-mini-instruct_Uncensored-Q4_K_M.gguf" --local-dir ./

If the model is split into multiple files (models larger than 50GB), you can download all the relevant files using:

huggingface-cli download bartowski/Phi-3.5-mini-instruct_Uncensored-GGUF --include "Phi-3.5-mini-instruct_Uncensored-Q8_0/*" --local-dir ./

Run the Model

You can run the quantized models in LM Studio.

✨ Features

• Multiple Quantization Types: Offers a wide range of quantization types (e.g., f16, Q8_0, Q6_K_L, etc.) to meet different resource and quality requirements.
• Embed/Output Weights Optimization: Some quantizations use Q8_0 for embeddings and output weights, potentially improving model quality.

📦 Installation

Install huggingface-cli

pip install -U "huggingface_hub[cli]"

💻 Usage Examples

Prompt Format

<s><|system|> {system_prompt}<|end|><|user|> {prompt}<|end|><|assistant|><|end|>

📚 Documentation

Model Information

Model Download Table

Embed/Output Weights

Some of these quants (Q3_K_XL, Q4_K_L etc) use the standard quantization method with the embeddings and output weights quantized to Q8_0 instead of the normal default. There is a debate on whether this improves the quality. If you use these models, please comment with your findings.

Model Selection Guide

A great write - up with charts showing various performances is provided by Artefact2 here.

To choose a model, first determine how much RAM and/or VRAM you have. If you want the model to run as fast as possible, choose a quant with a file size 1 - 2GB smaller than your GPU's total VRAM. If you want the maximum quality, add your system RAM and GPU's VRAM together and choose a quant 1 - 2GB smaller than that total.

You also need to decide between 'I - quants' (e.g., IQ3_M) and 'K - quants' (e.g., Q5_K_M). If you don't want to think too much, choose a K - quant. If you're aiming for below Q4 and running cuBLAS (Nvidia) or rocBLAS (AMD), consider I - quants, which are newer and offer better performance for their size. Note that I - quants are not compatible with Vulcan.

🔧 Technical Details

• Quantization Method: Uses llama.cpp release b3600 for quantization.
• Calibration Dataset: All quants are made using the imatrix option with the dataset from here.

📄 License

This project is licensed under the apache - 2.0 license.

👨‍💻 Credits

• Thank you kalomaze and Dampf for assistance in creating the imatrix calibration dataset.
• Thank you ZeroWw for the inspiration to experiment with embed/output.

💡 Usage Tip

If you want to support the developer's work, visit the ko - fi page: https://ko-fi.com/bartowski