Microsoft Phi 4 Reasoning GGUF

🚀 Llamacpp imatrix Quantizations of Phi-4-reasoning by microsoft

This project offers Llama.cpp imatrix quantizations of the Phi-4-reasoning model by Microsoft. It provides various quantization types to balance between model quality and resource usage, enabling users to run the model efficiently on different platforms.

🚀 Quick Start

• Using llama.cpp: Use llama.cpp release b5228 for quantization.
• Running the Model: You can run the quantized models in LM Studio or directly with llama.cpp, or any other llama.cpp based project.

✨ Features

• Multiple Quantization Types: Offers a wide range of quantization types, such as bf16, Q8_0, Q6_K_L, etc., to meet different requirements for model quality and resource consumption.
• Embed/Output Weights Optimization: Some quantizations (e.g., Q3_K_XL, Q4_K_L) use Q8_0 for embed and output weights, improving model performance.
• Online Repacking: Supports online repacking of weights for ARM and AVX machines, enhancing performance.

📦 Installation

Installing huggingface-cli

First, make sure you have hugginface-cli installed:

pip install -U "huggingface_hub[cli]"

Downloading a Specific File

Then, you can target the specific file you want:

huggingface-cli download bartowski/microsoft_Phi-4-reasoning-GGUF --include "microsoft_Phi-4-reasoning-Q4_K_M.gguf" --local-dir ./

Downloading Split Files

If the model is bigger than 50GB, it will have been split into multiple files. In order to download them all to a local folder, run:

huggingface-cli download bartowski/microsoft_Phi-4-reasoning-GGUF --include "microsoft_Phi-4-reasoning-Q8_0/*" --local-dir ./

You can either specify a new local-dir (microsoft_Phi-4-reasoning-Q8_0) or download them all in place (./)

💻 Usage Examples

Prompt Format

<|im_start|>system<|im_sep|>You are Phi, a language model trained by Microsoft to help users. Your role as an assistant involves thoroughly exploring questions through a systematic thinking process before providing the final precise and accurate solutions. This requires engaging in a comprehensive cycle of analysis, summarizing, exploration, reassessment, reflection, backtracing, and iteration to develop well-considered thinking process. Please structure your response into two main sections: Thought and Solution using the specified format:<think>{Thought section}</think>{Solution section}. In the Thought section, detail your reasoning process in steps. Each step should include detailed considerations such as analysing questions, summarizing relevant findings, brainstorming new ideas, verifying the accuracy of the current steps, refining any errors, and revisiting previous steps. In the Solution section, based on various attempts, explorations, and reflections from the Thought section, systematically present the final solution that you deem correct. The Solution section should be logical, accurate, and concise and detail necessary steps needed to reach the conclusion. Now, try to solve the following question through the above guidelines:<|im_end|>{system_prompt}<|end|><|user|>{prompt}<|end|><|assistant|>

📚 Documentation

Downloadable Files

Embed/Output Weights

Some of these quants (Q3_K_XL, Q4_K_L etc) are the standard quantization method with the embeddings and output weights quantized to Q8_0 instead of what they would normally default to.

ARM/AVX Information

Previously, you would download Q4_0_4_4/4_8/8_8, and these would have their weights interleaved in memory in order to improve performance on ARM and AVX machines by loading up more data in one pass.

Now, however, there is something called "online repacking" for weights. details in this PR. If you use Q4_0 and your hardware would benefit from repacking weights, it will do it automatically on the fly.

As of llama.cpp build b4282 you will not be able to run the Q4_0_X_X files and will instead need to use Q4_0.

Additionally, if you want to get slightly better quality for , you can use IQ4_NL thanks to this PR which will also repack the weights for ARM, though only the 4_4 for now. The loading time may be slower but it will result in an overall speed incrase.