Mythalion 13B GGUF

🚀 Mythalion 13B - GGUF

This repository provides GGUF format model files for Mythalion 13B, a text - generation model. It offers various quantized models suitable for different use - cases and hardware setups.

🚀 Quick Start

Downloading the Model

• Using Clients/Libraries: Tools like LM Studio, LoLLMS Web UI, and Faraday.dev can automatically download models. They present a list of available models for you to choose from.
• In text - generation - webui: Under "Download Model", enter the model repo TheBloke/Mythalion - 13B - GGUF and specify a filename (e.g., mythalion - 13b.q4_K_M.gguf), then click "Download".
• On the Command Line:
  • First, install the huggingface - hub Python library:

pip3 install huggingface - hub>=0.17.1

- Then download an individual model file:

huggingface - cli download TheBloke/Mythalion - 13B - GGUF mythalion - 13b.q4_K_M.gguf --local - dir. --local - dir - use - symlinks False

Running the Model

Example llama.cpp command

Ensure you are using llama.cpp from commit d0cee0d36d5be95a0d9088b674dbb27354107221 or later.

./main -ngl 32 -m mythalion - 13b.q4_K_M.gguf --color -c 4096 --temp 0.7 --repeat_penalty 1.1 -n -1 -p "Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n### Instruction:\n{prompt}\n\n### Response:"

• Adjust -ngl 32 according to the number of layers you want to offload to the GPU. Remove it if you don't have GPU acceleration.
• Modify -c 4096 to set the desired sequence length. For extended sequence models, llama.cpp automatically reads and sets the necessary RoPE scaling parameters from the GGUF file.
• To have a chat - style conversation, replace the -p <PROMPT> argument with -i -ins.

Running in text - generation - webui

Refer to [text - generation - webui/docs/llama.cpp.md](https://github.com/oobabooga/text - generation - webui/blob/main/docs/llama.cpp.md) for further instructions.

Running from Python code

You can use GGUF models from Python with the [llama - cpp - python](https://github.com/abetlen/llama - cpp - python) or ctransformers libraries.

✨ Features

• Multiple Quantization Options: Offers a range of quantized models (e.g., Q2_K, Q3_K, etc.) to balance between model size and quality.
• Broad Compatibility: Compatible with many clients and libraries such as llama.cpp, text - generation - webui, KoboldCpp, etc.
• Flexible Usage: Can be used for both instruction - following tasks and chat - style conversations.

📦 Installation

Installing Dependencies for Download

To download models on the command line, you need to install the huggingface - hub Python library:

pip3 install huggingface - hub>=0.17.1

To accelerate downloads on fast connections, install hf_transfer:

pip3 install hf_transfer

Installing Dependencies for Python Usage

If you want to use the model from Python:

• For ctransformers without GPU acceleration:

pip install ctransformers>=0.2.24

• With CUDA GPU acceleration:

pip install ctransformers[cuda]>=0.2.24

• With ROCm GPU acceleration:

CT_HIPBLAS = 1 pip install ctransformers

💻 Usage Examples

Basic Usage in llama.cpp

./main -ngl 32 -m mythalion - 13b.q4_K_M.gguf --color -c 4096 --temp 0.7 --repeat_penalty 1.1 -n -1 -p "Below is an instruction that describes a task. Write a response that appropriately completes the request.\n\n### Instruction:\n{prompt}\n\n### Response:"

Advanced Usage - Chat - style Conversation in llama.cpp

./main -ngl 32 -m mythalion - 13b.q4_K_M.gguf --color -c 4096 --temp 0.7 --repeat_penalty 1.1 -n -1 -i -ins

📚 Documentation

About GGUF

GGUF is a new format introduced by the llama.cpp team on August 21st, 2023. It replaces GGML, which is no longer supported by llama.cpp. GGUF has several advantages over GGML, including better tokenization, support for special tokens, metadata support, and extensibility.

Here is a list of clients and libraries known to support GGUF:

• llama.cpp: The source project for GGUF, offering a CLI and a server option.
• [text - generation - webui](https://github.com/oobabooga/text - generation - webui): A widely used web UI with many features and powerful extensions, supporting GPU acceleration.
• KoboldCpp: A fully - featured web UI with GPU acceleration across all platforms and GPU architectures, great for story - telling.
• LM Studio: An easy - to - use and powerful local GUI for Windows and macOS (Silicon), with GPU acceleration.
• [LoLLMS Web UI](https://github.com/ParisNeo/lollms - webui): A web UI with many interesting and unique features, including a full model library for easy model selection.
• Faraday.dev: An attractive and easy - to - use character - based chat GUI for Windows and macOS (both Silicon and Intel), with GPU acceleration.
• ctransformers: A Python library with GPU acceleration, LangChain support, and an OpenAI - compatible AI server.
• [llama - cpp - python](https://github.com/abetlen/llama - cpp - python): A Python library with GPU acceleration, LangChain support, and an OpenAI - compatible API server.
• candle: A Rust ML framework focusing on performance, including GPU support and ease of use.

Repositories available

• [AWQ model(s) for GPU inference.](https://huggingface.co/TheBloke/Mythalion - 13B - AWQ)
• [GPTQ models for GPU inference, with multiple quantisation parameter options.](https://huggingface.co/TheBloke/Mythalion - 13B - GPTQ)
• [2, 3, 4, 5, 6 and 8 - bit GGUF models for CPU+GPU inference](https://huggingface.co/TheBloke/Mythalion - 13B - GGUF)
• [PygmalionAI's original unquantised fp16 model in pytorch format, for GPU inference and for further conversions](https://huggingface.co/PygmalionAI/mythalion - 13b)

Prompt template

Below is an instruction that describes a task. Write a response that appropriately completes the request.

### Instruction:
{prompt}

### Response:

Compatibility

These quantised GGUFv2 files are compatible with llama.cpp from August 27th onwards, as of commit d0cee0d36d5be95a0d9088b674dbb27354107221. They are also compatible with many third - party UIs and libraries; refer to the list at the top of this README.

Explanation of quantisation methods

Provided files

Note: the above RAM figures assume no GPU offloading. If layers are offloaded to the GPU, this will reduce RAM usage and use VRAM instead.

🔧 Technical Details

The GGUF format is designed to be more efficient and feature - rich compared to the deprecated GGML format. It allows for better tokenization, support for special tokens, and metadata storage. Different quantisation methods are used to balance between model size and quality. For example, lower - bit quantisation methods like Q2_K result in smaller model sizes but may have significant quality loss, while higher - bit methods like Q6_K and Q8_0 have extremely low quality loss but larger model sizes.

📄 License

The model is licensed under the llama2 license.