Model Overview
Model Features
Model Capabilities
Use Cases
đ Yi 6B - GGUF
This repository provides GGUF format model files for 01-ai's Yi 6B, offering efficient and flexible options for text generation.
đ Quick Start
This repo contains GGUF format model files for 01-ai's Yi 6B. These files were quantised using hardware kindly provided by Massed Compute.
⨠Features
- New GGUF Format: GGUF is a new format introduced by the llama.cpp team on August 21st 2023, replacing the no-longer-supported GGML.
- Multiple Compatibilities: Compatible with various clients and libraries such as llama.cpp, text-generation-webui, KoboldCpp, etc.
- Diverse Quantization Options: Offers multiple quantization methods and corresponding files to meet different needs in terms of size, quality, and RAM usage.
đĻ Installation
How to download GGUF files
Note for manual downloaders: You almost never want to clone the entire repo! Multiple different quantisation formats are provided, and most users only want to pick and download a single file.
The following clients/libraries will automatically download models for you, providing a list of available models to choose from:
- LM Studio
- LoLLMS Web UI
- Faraday.dev
In text-generation-webui
Under Download Model, you can enter the model repo: TheBloke/Yi-6B-GGUF and below it, a specific filename to download, such as: yi-6b.Q4_K_M.gguf. Then click Download.
On the command line, including multiple files at once
I recommend using the huggingface-hub
Python library:
pip3 install huggingface-hub
Then you can download any individual model file to the current directory, at high speed, with a command like this:
huggingface-cli download TheBloke/Yi-6B-GGUF yi-6b.Q4_K_M.gguf --local-dir . --local-dir-use-symlinks False
More advanced huggingface-cli download usage
You can also download multiple files at once with a pattern:
huggingface-cli download TheBloke/Yi-6B-GGUF --local-dir . --local-dir-use-symlinks False --include='*Q4_K*gguf'
For more documentation on downloading with huggingface-cli
, please see: HF -> Hub Python Library -> Download files -> Download from the CLI.
To accelerate downloads on fast connections (1Gbit/s or higher), install hf_transfer
:
pip3 install hf_transfer
And set environment variable HF_HUB_ENABLE_HF_TRANSFER
to 1
:
HF_HUB_ENABLE_HF_TRANSFER=1 huggingface-cli download TheBloke/Yi-6B-GGUF yi-6b.Q4_K_M.gguf --local-dir . --local-dir-use-symlinks False
Windows Command Line users: You can set the environment variable by running set HF_HUB_ENABLE_HF_TRANSFER=1
before the download command.
đģ Usage Examples
Example llama.cpp
command
Make sure you are using llama.cpp
from commit d0cee0d or later.
./main -ngl 32 -m yi-6b.Q4_K_M.gguf --color -c 2048 --temp 0.7 --repeat_penalty 1.1 -n -1 -p "{prompt}"
Change -ngl 32
to the number of layers to offload to GPU. Remove it if you don't have GPU acceleration.
Change -c 2048
to the desired sequence length. For extended sequence models - eg 8K, 16K, 32K - the necessary RoPE scaling parameters are read from the GGUF file and set by llama.cpp automatically.
If you want to have a chat-style conversation, replace the -p <PROMPT>
argument with -i -ins
For other parameters and how to use them, please refer to the llama.cpp documentation
How to run in text-generation-webui
Further instructions here: text-generation-webui/docs/llama.cpp.md.
How to run from Python code
You can use GGUF models from Python using the llama-cpp-python or ctransformers libraries.
How to load this model in Python code, using ctransformers
First install the package
Run one of the following commands, according to your system:
# Base ctransformers with no GPU acceleration
pip install ctransformers
# Or with CUDA GPU acceleration
pip install ctransformers[cuda]
# Or with AMD ROCm GPU acceleration (Linux only)
CT_HIPBLAS=1 pip install ctransformers --no-binary ctransformers
# Or with Metal GPU acceleration for macOS systems only
CT_METAL=1 pip install ctransformers --no-binary ctransformers
Simple ctransformers example code
from ctransformers import AutoModelForCausalLM
# Set gpu_layers to the number of layers to offload to GPU. Set to 0 if no GPU acceleration is available on your system.
llm = AutoModelForCausalLM.from_pretrained("TheBloke/Yi-6B-GGUF", model_file="yi-6b.Q4_K_M.gguf", model_type="yi", gpu_layers=50)
print(llm("AI is going to"))
How to use with LangChain
Here are guides on using llama-cpp-python and ctransformers with LangChain:
đ Documentation
About GGUF
GGUF is a new format introduced by the llama.cpp team on August 21st 2023. It is a replacement for GGML, which is no longer supported by llama.cpp.
Here is an incomplete list of clients and libraries that are known to support GGUF:
- llama.cpp. The source project for GGUF. Offers a CLI and a server option.
- text-generation-webui, the most widely used web UI, with many features and powerful extensions. Supports GPU acceleration.
- KoboldCpp, a fully featured web UI, with GPU accel across all platforms and GPU architectures. Especially good for story telling.
- LM Studio, an easy-to-use and powerful local GUI for Windows and macOS (Silicon), with GPU acceleration.
- LoLLMS Web UI, a great web UI with many interesting and unique features, including a full model library for easy model selection.
- Faraday.dev, an attractive and easy to use character-based chat GUI for Windows and macOS (both Silicon and Intel), with GPU acceleration.
- ctransformers, a Python library with GPU accel, LangChain support, and OpenAI-compatible AI server.
- llama-cpp-python, a Python library with GPU accel, LangChain support, and OpenAI-compatible API server.
- candle, a Rust ML framework with a focus on performance, including GPU support, and ease of use.
Repositories available
- AWQ model(s) for GPU inference.
- GPTQ models for GPU inference, with multiple quantisation parameter options.
- 2, 3, 4, 5, 6 and 8 - bit GGUF models for CPU+GPU inference
- 01-ai's original unquantised fp16 model in pytorch format, for GPU inference and for further conversions
Prompt template: None
{prompt}
Compatibility
These quantised GGUFv2 files are compatible with llama.cpp from August 27th onwards, as of commit d0cee0d They are also compatible with many third party UIs and libraries - please see the list at the top of this README.
Explanation of quantisation methods
Click to see details
The new methods available are:
- GGML_TYPE_Q2_K - "type - 1" 2 - bit quantization in super - blocks containing 16 blocks, each block having 16 weight. Block scales and mins are quantized with 4 bits. This ends up effectively using 2.5625 bits per weight (bpw)
- GGML_TYPE_Q3_K - "type - 0" 3 - bit quantization in super - blocks containing 16 blocks, each block having 16 weights. Scales are quantized with 6 bits. This end up using 3.4375 bpw.
- GGML_TYPE_Q4_K - "type - 1" 4 - bit quantization in super - blocks containing 8 blocks, each block having 32 weights. Scales and mins are quantized with 6 bits. This ends up using 4.5 bpw.
- GGML_TYPE_Q5_K - "type - 1" 5 - bit quantization. Same super - block structure as GGML_TYPE_Q4_K resulting in 5.5 bpw
- GGML_TYPE_Q6_K - "type - 0" 6 - bit quantization. Super - blocks with 16 blocks, each block having 16 weights. Scales are quantized with 8 bits. This ends up using 6.5625 bpw
Refer to the Provided Files table below to see what files use which methods, and how.
Provided files
Name | Quant method | Bits | Size | Max RAM required | Use case |
---|---|---|---|---|---|
yi-6b.Q2_K.gguf | Q2_K | 2 | 2.62 GB | 5.12 GB | smallest, significant quality loss - not recommended for most purposes |
yi-6b.Q3_K_S.gguf | Q3_K_S | 3 | 2.71 GB | 5.21 GB | very small, high quality loss |
yi-6b.Q3_K_M.gguf | Q3_K_M | 3 | 2.99 GB | 5.49 GB | very small, high quality loss |
yi-6b.Q3_K_L.gguf | Q3_K_L | 3 | 3.24 GB | 5.74 GB | small, substantial quality loss |
yi-6b.Q4_0.gguf | Q4_0 | 4 | 3.48 GB | 5.98 GB | legacy; small, very high quality loss - prefer using Q3_K_M |
yi-6b.Q4_K_S.gguf | Q4_K_S | 4 | 3.50 GB | 6.00 GB | small, greater quality loss |
yi-6b.Q4_K_M.gguf | Q4_K_M | 4 | 3.67 GB | 6.17 GB | medium, balanced quality - recommended |
yi-6b.Q5_0.gguf | Q5_0 | 5 | 4.20 GB | 6.70 GB | legacy; medium, balanced quality - prefer using Q4_K_M |
yi-6b.Q5_K_S.gguf | Q5_K_S | 5 | 4.20 GB | 6.70 GB | large, low quality loss - recommended |
yi-6b.Q5_K_M.gguf | Q5_K_M | 5 | 4.30 GB | 6.80 GB | large, very low quality loss - recommended |
yi-6b.Q6_K.gguf | Q6_K | 6 | 4.97 GB | 7.47 GB | very large, extremely low quality loss |
yi-6b.Q8_0.gguf | Q8_0 | 8 | 6.44 GB | 8.94 GB | very large, extremely low quality loss - not recommended |
Note: the above RAM figures assume no GPU offloading. If layers are offloaded to the GPU, this will reduce RAM usage and use VRAM instead.
đ License
The model is under the yi - license.
Discord
For further support, and discussions on these models and AI in general, join us at: TheBloke AI's Discord server

