Mixtral 8x7B V0.1 GGUF

🚀 Mixtral-8x7B-v0.1-GGUF

This project provides GGUF format model files for mistralai/Mixtral-8x7B-v0.1, enabling efficient use of the model in various applications.

✨ Features

• Quantized Variants: Available in 2-bit, 3-bit, 4-bit, 5-bit, 6-bit, and 8-bit quantization.
• Multiple Languages Support: Supports multiple languages including French, Italian, German, Spanish, and English.
• Compatibility: Compatible with autotrain, endpoints, and various clients/libraries like llama.cpp, text-generation-webui, etc.

📦 Installation

Install huggingface-hub

pip3 install huggingface-hub

Install hf_transfer for Faster Downloads

pip3 install hf_transfer

💻 Usage Examples

Basic Usage in llama.cpp

./main -ngl 35 -m Mixtral-8x7B-v0.1-GGUF.Q4_K_M.gguf --color -c 32768 --temp 0.7 --repeat_penalty 1.1 -n -1 -p "<|im_start|>system
{system_message}<|im_end|>
<|im_start|>user
{prompt}<|im_end|>
<|im_start|>assistant"

Advanced Usage in Python with llama-cpp-python

from llama_cpp import Llama

# Set gpu_layers to the number of layers to offload to GPU. Set to 0 if no GPU acceleration is available on your system.
llm = Llama(
  model_path="./Mixtral-8x7B-v0.1-GGUF.Q4_K_M.gguf",  # Download the model file first
  n_ctx=32768,  # The max sequence length to use - note that longer sequence lengths require much more resources
  n_threads=8,            # The number of CPU threads to use, tailor to your system and the resulting performance
  n_gpu_layers=35         # The number of layers to offload to GPU, if you have GPU acceleration available
)

# Simple inference example
output = llm(
  "<|im_start|>system
{system_message}<|im_end|>
<|im_start|>user
{prompt}<|im_end|>
<|im_start|>assistant", # Prompt
  max_tokens=512,  # Generate up to 512 tokens
  stop=["</s>"],   # Example stop token - not necessarily correct for this specific model! Please check before using.
  echo=True        # Whether to echo the prompt
)

# Chat Completion API

llm = Llama(model_path="./Mixtral-8x7B-v0.1-GGUF.Q4_K_M.gguf", chat_format="llama-2")  # Set chat_format according to the model you are using
llm.create_chat_completion(
    messages = [
        {"role": "system", "content": "You are a story writing assistant."},
        {
            "role": "user",
            "content": "Write a story about llamas."
        }
    ]
)

📚 Documentation

About GGUF

GGUF is a new format introduced by the llama.cpp team on August 21st 2023. It is a replacement for GGML, which is no longer supported by llama.cpp.

Here is an incomplete list of clients and libraries that are known to support GGUF:

• llama.cpp. The source project for GGUF. Offers a CLI and a server option.
• text-generation-webui, the most widely used web UI, with many features and powerful extensions. Supports GPU acceleration.
• KoboldCpp, a fully featured web UI, with GPU accel across all platforms and GPU architectures. Especially good for story telling.
• GPT4All, a free and open source local running GUI, supporting Windows, Linux and macOS with full GPU accel.
• LM Studio, an easy-to-use and powerful local GUI for Windows and macOS (Silicon), with GPU acceleration. Linux available, in beta as of 27/11/2023.
• LoLLMS Web UI, a great web UI with many interesting and unique features, including a full model library for easy model selection.
• Faraday.dev, an attractive and easy to use character-based chat GUI for Windows and macOS (both Silicon and Intel), with GPU acceleration.
• llama-cpp-python, a Python library with GPU accel, LangChain support, and OpenAI-compatible API server.
• candle, a Rust ML framework with a focus on performance, including GPU support, and ease of use.
• ctransformers, a Python library with GPU accel, LangChain support, and OpenAI-compatible AI server. Note, as of time of writing (November 27th 2023), ctransformers has not been updated in a long time and does not support many recent models.

Explanation of quantisation methods

How to download GGUF files

Note for manual downloaders: You almost never want to clone the entire repo! Multiple different quantisation formats are provided, and most users only want to pick and download a single file.

The following clients/libraries will automatically download models for you, providing a list of available models to choose from:

• LM Studio
• LoLLMS Web UI
• Faraday.dev

In text-generation-webui

Under Download Model, you can enter the model repo: MaziyarPanahi/Mixtral-8x7B-v0.1-GGUF and below it, a specific filename to download, such as: Mixtral-8x7B-v0.1-GGUF.Q4_K_M.gguf. Then click Download.

On the command line, including multiple files at once

You can download any individual model file to the current directory, at high speed, with a command like this:

huggingface-cli download MaziyarPanahi/Mixtral-8x7B-v0.1-GGUF Mixtral-8x7B-v0.1-GGUF.Q4_K_M.gguf --local-dir . --local-dir-use-symlinks False

You can also download multiple files at once with a pattern:

huggingface-cli download [MaziyarPanahi/Mixtral-8x7B-v0.1-GGUF](https://huggingface.co/MaziyarPanahi/Mixtral-8x7B-v0.1-GGUF) --local-dir . --local-dir-use-symlinks False --include='*Q4_K*gguf'

For more documentation on downloading with huggingface-cli, please see: HF -> Hub Python Library -> Download files -> Download from the CLI.

To accelerate downloads on fast connections (1Gbit/s or higher), set environment variable HF_HUB_ENABLE_HF_TRANSFER to 1:

HF_HUB_ENABLE_HF_TRANSFER=1 huggingface-cli download MaziyarPanahi/Mixtral-8x7B-v0.1-GGUF Mixtral-8x7B-v0.1-GGUF.Q4_K_M.gguf --local-dir . --local-dir-use-symlinks False

Windows Command Line users: You can set the environment variable by running set HF_HUB_ENABLE_HF_TRANSFER=1 before the download command.

How to run in text-generation-webui

Further instructions can be found in the text-generation-webui documentation, here: text-generation-webui/docs/04 ‐ Model Tab.md.

How to use with LangChain

Here are guides on using llama-cpp-python and ctransformers with LangChain:

• LangChain + llama-cpp-python
• LangChain + ctransformers

📄 License

This project is licensed under the Apache-2.0 license.