Llongorca 7B 16K GGUF

🚀 LlongOrca 7B 16K - GGUF

This repository provides GGUF format model files for the LlongOrca 7B 16K model, offering various quantized versions for different use - cases.

🚀 Quick Start

This README offers comprehensive information about the LlongOrca 7B 16K model in GGUF format, including details about the model, quantisation methods, compatibility, and how to download and run the model.

✨ Features

• Multiple Quantization Options: Different quantization methods are available to balance between model size and quality.
• Broad Compatibility: Compatible with many popular clients, libraries, and UIs such as llama.cpp, text - generation - webui, etc.
• Extended Sequence Support: Capable of handling extended sequence lengths.

📦 Installation

How to download GGUF files

Note for manual downloaders: You almost never want to clone the entire repo! Multiple different quantisation formats are provided, and most users only want to pick and download a single file.

The following clients/libraries will automatically download models for you, providing a list of available models to choose from:

• LM Studio
• LoLLMS Web UI
• Faraday.dev

In text - generation - webui

Under Download Model, you can enter the model repo: TheBloke/LlongOrca - 7B - 16K - GGUF and below it, a specific filename to download, such as: llongorca - 7b - 16k.q4_K_M.gguf. Then click Download.

On the command line, including multiple files at once

I recommend using the huggingface - hub Python library:

pip3 install huggingface - hub>=0.17.1

Then you can download any individual model file to the current directory, at high speed, with a command like this:

huggingface - cli download TheBloke/LlongOrca - 7B - 16K - GGUF llongorca - 7b - 16k.q4_K_M.gguf --local - dir. --local - dir - use - symlinks False

pip3 install hf_transfer

HUGGINGFACE_HUB_ENABLE_HF_TRANSFER = 1 huggingface - cli download TheBloke/LlongOrca - 7B - 16K - GGUF llongorca - 7b - 16k.q4_K_M.gguf --local - dir. --local - dir - use - symlinks False

💻 Usage Examples

Basic Usage

Example llama.cpp command

Make sure you are using llama.cpp from commit d0cee0d36d5be95a0d9088b674dbb27354107221 or later.

./main -ngl 32 -m llongorca - 7b - 16k.q4_K_M.gguf --color -c 4096 --temp 0.7 --repeat_penalty 1.1 -n -1 -p "<|im_start|>system\n{system_message}<|im_end|>\n<|im_start|>user\n{prompt}<|im_end|>\n<|im_start|>assistant"

Change -ngl 32 to the number of layers to offload to GPU. Remove it if you don't have GPU acceleration.

Change -c 4096 to the desired sequence length. For extended sequence models - eg 8K, 16K, 32K - the necessary RoPE scaling parameters are read from the GGUF file and set by llama.cpp automatically.

If you want to have a chat - style conversation, replace the -p <PROMPT> argument with -i -ins

For other parameters and how to use them, please refer to the llama.cpp documentation

Advanced Usage

How to run in text - generation - webui

Further instructions here: [text - generation - webui/docs/llama.cpp.md](https://github.com/oobabooga/text - generation - webui/blob/main/docs/llama.cpp.md).

How to run from Python code

You can use GGUF models from Python using the [llama - cpp - python](https://github.com/abetlen/llama - cpp - python) or ctransformers libraries.

How to load this model from Python using ctransformers

First install the package

# Base ctransformers with no GPU acceleration
pip install ctransformers>=0.2.24
# Or with CUDA GPU acceleration
pip install ctransformers[cuda]>=0.2.24
# Or with ROCm GPU acceleration
CT_HIPBLAS = 1 pip install ctransformers>=0.2.24 --no - binary ctransformers
# Or with Metal GPU acceleration for macOS systems
CT_METAL = 1 pip install ctransformers

📚 Documentation

About the Model

• Model Creator: [Open - Orca](https://huggingface.co/Open - Orca)
• Original Model: [LlongOrca 7B 16K](https://huggingface.co/Open - Orca/LlongOrca - 7B - 16k)
• Model Type: llama
• Pipeline Tag: text - generation

Prompt Template

<|im_start|>system
{system_message}<|im_end|>
<|im_start|>user
{prompt}<|im_end|>
<|im_start|>assistant

Repositories available

• [AWQ model(s) for GPU inference.](https://huggingface.co/TheBloke/LlongOrca - 7B - 16K - AWQ)
• [GPTQ models for GPU inference, with multiple quantisation parameter options.](https://huggingface.co/TheBloke/LlongOrca - 7B - 16K - GPTQ)
• [2, 3, 4, 5, 6 and 8 - bit GGUF models for CPU+GPU inference](https://huggingface.co/TheBloke/LlongOrca - 7B - 16K - GGUF)
• [Open - Orca's original unquantised fp16 model in pytorch format, for GPU inference and for further conversions](https://huggingface.co/Open - Orca/LlongOrca - 7B - 16k)

Compatibility

These quantised GGUFv2 files are compatible with llama.cpp from August 27th onwards, as of commit d0cee0d36d5be95a0d9088b674dbb27354107221

They are also compatible with many third - party UIs and libraries - please see the list at the top of this README.

Explanation of quantisation methods

Provided files

Note: the above RAM figures assume no GPU offloading. If layers are offloaded to the GPU, this will reduce RAM usage and use VRAM instead.

🔧 Technical Details

• Model Type: llama
• Training Data: Open - Orca/OpenOrca

📄 License

The model is under the llama2 license.