Yi 6B GGUF

🚀 Yi 6B - GGUF

This repository provides GGUF format model files for 01-ai's Yi 6B, offering efficient and flexible options for text generation.

🚀 Quick Start

This repo contains GGUF format model files for 01-ai's Yi 6B. These files were quantised using hardware kindly provided by Massed Compute.

✨ Features

• New GGUF Format: GGUF is a new format introduced by the llama.cpp team on August 21st 2023, replacing the no-longer-supported GGML.
• Multiple Compatibilities: Compatible with various clients and libraries such as llama.cpp, text-generation-webui, KoboldCpp, etc.
• Diverse Quantization Options: Offers multiple quantization methods and corresponding files to meet different needs in terms of size, quality, and RAM usage.

📦 Installation

How to download GGUF files

Note for manual downloaders: You almost never want to clone the entire repo! Multiple different quantisation formats are provided, and most users only want to pick and download a single file.

The following clients/libraries will automatically download models for you, providing a list of available models to choose from:

• LM Studio
• LoLLMS Web UI
• Faraday.dev

In text-generation-webui

Under Download Model, you can enter the model repo: TheBloke/Yi-6B-GGUF and below it, a specific filename to download, such as: yi-6b.Q4_K_M.gguf. Then click Download.

On the command line, including multiple files at once

I recommend using the huggingface-hub Python library:

pip3 install huggingface-hub

Then you can download any individual model file to the current directory, at high speed, with a command like this:

huggingface-cli download TheBloke/Yi-6B-GGUF yi-6b.Q4_K_M.gguf --local-dir . --local-dir-use-symlinks False

huggingface-cli download TheBloke/Yi-6B-GGUF --local-dir . --local-dir-use-symlinks False --include='*Q4_K*gguf'

pip3 install hf_transfer

HF_HUB_ENABLE_HF_TRANSFER=1 huggingface-cli download TheBloke/Yi-6B-GGUF yi-6b.Q4_K_M.gguf --local-dir . --local-dir-use-symlinks False

💻 Usage Examples

Example llama.cpp command

Make sure you are using llama.cpp from commit d0cee0d or later.

./main -ngl 32 -m yi-6b.Q4_K_M.gguf --color -c 2048 --temp 0.7 --repeat_penalty 1.1 -n -1 -p "{prompt}"

Change -ngl 32 to the number of layers to offload to GPU. Remove it if you don't have GPU acceleration. Change -c 2048 to the desired sequence length. For extended sequence models - eg 8K, 16K, 32K - the necessary RoPE scaling parameters are read from the GGUF file and set by llama.cpp automatically. If you want to have a chat-style conversation, replace the -p <PROMPT> argument with -i -ins For other parameters and how to use them, please refer to the llama.cpp documentation

How to run in text-generation-webui

Further instructions here: text-generation-webui/docs/llama.cpp.md.

How to run from Python code

You can use GGUF models from Python using the llama-cpp-python or ctransformers libraries.

How to load this model in Python code, using ctransformers

First install the package

Run one of the following commands, according to your system:

# Base ctransformers with no GPU acceleration
pip install ctransformers
# Or with CUDA GPU acceleration
pip install ctransformers[cuda]
# Or with AMD ROCm GPU acceleration (Linux only)
CT_HIPBLAS=1 pip install ctransformers --no-binary ctransformers
# Or with Metal GPU acceleration for macOS systems only
CT_METAL=1 pip install ctransformers --no-binary ctransformers

Simple ctransformers example code

from ctransformers import AutoModelForCausalLM

# Set gpu_layers to the number of layers to offload to GPU. Set to 0 if no GPU acceleration is available on your system.
llm = AutoModelForCausalLM.from_pretrained("TheBloke/Yi-6B-GGUF", model_file="yi-6b.Q4_K_M.gguf", model_type="yi", gpu_layers=50)

print(llm("AI is going to"))

How to use with LangChain

Here are guides on using llama-cpp-python and ctransformers with LangChain:

• LangChain + llama-cpp-python
• LangChain + ctransformers

📚 Documentation

About GGUF

GGUF is a new format introduced by the llama.cpp team on August 21st 2023. It is a replacement for GGML, which is no longer supported by llama.cpp.

Here is an incomplete list of clients and libraries that are known to support GGUF:

• llama.cpp. The source project for GGUF. Offers a CLI and a server option.
• text-generation-webui, the most widely used web UI, with many features and powerful extensions. Supports GPU acceleration.
• KoboldCpp, a fully featured web UI, with GPU accel across all platforms and GPU architectures. Especially good for story telling.
• LM Studio, an easy-to-use and powerful local GUI for Windows and macOS (Silicon), with GPU acceleration.
• LoLLMS Web UI, a great web UI with many interesting and unique features, including a full model library for easy model selection.
• Faraday.dev, an attractive and easy to use character-based chat GUI for Windows and macOS (both Silicon and Intel), with GPU acceleration.
• ctransformers, a Python library with GPU accel, LangChain support, and OpenAI-compatible AI server.
• llama-cpp-python, a Python library with GPU accel, LangChain support, and OpenAI-compatible API server.
• candle, a Rust ML framework with a focus on performance, including GPU support, and ease of use.

Repositories available

• AWQ model(s) for GPU inference.
• GPTQ models for GPU inference, with multiple quantisation parameter options.
• 2, 3, 4, 5, 6 and 8 - bit GGUF models for CPU+GPU inference
• 01-ai's original unquantised fp16 model in pytorch format, for GPU inference and for further conversions

Prompt template: None

{prompt}

Compatibility

These quantised GGUFv2 files are compatible with llama.cpp from August 27th onwards, as of commit d0cee0d They are also compatible with many third party UIs and libraries - please see the list at the top of this README.

Explanation of quantisation methods

Provided files

Note: the above RAM figures assume no GPU offloading. If layers are offloaded to the GPU, this will reduce RAM usage and use VRAM instead.

📄 License

The model is under the yi - license.

Discord

For further support, and discussions on these models and AI in general, join us at: TheBloke AI's Discord server