Llama 3 Cat 8b Instruct V1 GGUF

🚀 Llamacpp imatrix Quantizations of llama-3-cat-8b-instruct-v1

This project focuses on the quantization of the llama-3-cat-8b-instruct-v1 model. It uses llama.cpp for quantization, providing various quantized versions with different file sizes and qualities to meet diverse user needs.

🚀 Quick Start

Prerequisites

Ensure you have huggingface-cli installed. You can install it using the following command:

pip install -U "huggingface_hub[cli]"

Download a Specific File

To download a specific file, use the following command. For example, to download llama-3-cat-8b-instruct-v1-Q4_K_M.gguf:

huggingface-cli download bartowski/llama-3-cat-8b-instruct-v1-GGUF --include "llama-3-cat-8b-instruct-v1-Q4_K_M.gguf" --local-dir ./ --local-dir-use-symlinks False

Download Split Files

If the model is larger than 50GB and split into multiple files, you can download all of them to a local folder using the following command:

huggingface-cli download bartowski/llama-3-cat-8b-instruct-v1-GGUF --include "llama-3-cat-8b-instruct-v1-Q8_0.gguf/*" --local-dir llama-3-cat-8b-instruct-v1-Q8_0 --local-dir-use-symlinks False

✨ Features

• Multiple Quantization Types: Offers a wide range of quantized models, including Q8_0, Q6_K, Q5_K_M, etc., to balance between quality and file size.
• Easy Download: Provides clear instructions on how to download files using huggingface-cli.
• Performance Guidance: Offers guidance on choosing the appropriate quantized file based on available RAM/VRAM and performance requirements.

📦 Installation

The main installation step is to install huggingface-cli using the command:

pip install -U "huggingface_hub[cli]"

💻 Usage Examples

Download a Specific File

huggingface-cli download bartowski/llama-3-cat-8b-instruct-v1-GGUF --include "llama-3-cat-8b-instruct-v1-Q4_K_M.gguf" --local-dir ./ --local-dir-use-symlinks False

Download Split Files

huggingface-cli download bartowski/llama-3-cat-8b-instruct-v1-GGUF --include "llama-3-cat-8b-instruct-v1-Q8_0.gguf/*" --local-dir llama-3-cat-8b-instruct-v1-Q8_0 --local-dir-use-symlinks False

📚 Documentation

Prompt Format

<|begin_of_text|><|start_header_id|>system<|end_header_id|>

{system_prompt}<|eot_id|><|start_header_id|>user<|end_header_id|>

{prompt}<|eot_id|><|start_header_id|>assistant<|end_header_id|>

File Download Table

Which File to Choose

A great write - up with charts showing various performances is provided by Artefact2 here.

• Determine Available Resources: First, figure out how much RAM and/or VRAM you have. If you want the model to run as fast as possible, choose a quant with a file size 1 - 2GB smaller than your GPU's total VRAM. If you want the maximum quality, add your system RAM and GPU's VRAM together and choose a quant 1 - 2GB smaller than that total.
• Choose between 'I - quant' and 'K - quant': If you don't want to think too much, choose a K - quant (e.g., Q5_K_M). If you're aiming for below Q4 and using cuBLAS (Nvidia) or rocBLAS (AMD), consider I - quants (e.g., IQ3_M). Note that I - quants are not compatible with Vulcan.

🔧 Technical Details

• Quantization Tool: Using llama.cpp release b2854 for quantization.
• Original Model: https://huggingface.co/TheSkullery/llama-3-cat-8b-instruct-v1
• Quantization Option: All quants are made using the imatrix option with the dataset provided by Kalomaze here

📄 License

The model uses the llama3 license.

💡 Usage Tip

If you want to support the author's work, you can visit the ko - fi page: https://ko-fi.com/bartowski