Norocetacean 20B 10k GGUF

🚀 Norocetacean 20B 10K - GGUF

This repository contains GGUF format model files for Norocetacean 20B 10K, offering text - generation capabilities.

🖼️ Header

📋 Model Information

✨ Features

This model is based on the Llama architecture and is designed for text - generation tasks. It comes with different quantization options to suit various hardware and performance requirements.

📦 Repositories Available

• AWQ model(s) for GPU inference.
• GPTQ models for GPU inference, with multiple quantisation parameter options.
• 2, 3, 4, 5, 6 and 8 - bit GGUF models for CPU+GPU inference
• ddh0's original unquantised fp16 model in pytorch format, for GPU inference and for further conversions

💻 Prompt Template

Below is an instruction that describes a task. Write a response that appropriately completes the request.

### Instruction:
{prompt}

### Response:

📄 Licensing

The creator of the source model has listed its license as other, and this quantization uses the same license. As this model is based on Llama 2, it is also subject to the Meta Llama 2 license terms, and the license files for that are additionally included. Any questions regarding licensing should be directed to the original model repository: ddh0's Norocetacean 20B 10K.

🔧 Compatibility

These quantised GGUFv2 files are compatible with llama.cpp from August 27th onwards, as of commit d0cee0d. They are also compatible with many third - party UIs and libraries.

Explanation of Quantisation Methods

📋 Provided Files

Note: the above RAM figures assume no GPU offloading. If layers are offloaded to the GPU, this will reduce RAM usage and use VRAM instead.

📥 How to Download GGUF Files

Note for manual downloaders: You almost never want to clone the entire repo! Multiple different quantisation formats are provided, and most users only want to pick and download a single file.

The following clients/libraries will automatically download models for you, providing a list of available models to choose from:

• LM Studio
• LoLLMS Web UI
• Faraday.dev

In text - generation - webui

Under Download Model, you can enter the model repo: TheBloke/Norocetacean-20B-10k-GGUF and below it, a specific filename to download, such as: norocetacean-20b-10k.Q4_K_M.gguf. Then click Download.

On the command line, including multiple files at once

I recommend using the huggingface - hub Python library:

pip3 install huggingface - hub

Then you can download any individual model file to the current directory, at high speed, with a command like this:

huggingface - cli download TheBloke/Norocetacean-20B-10k-GGUF norocetacean-20b-10k.Q4_K_M.gguf --local - dir . --local - dir - use - symlinks False

huggingface - cli download TheBloke/Norocetacean-20B-10k-GGUF --local - dir . --local - dir - use - symlinks False --include='*Q4_K*gguf'

pip3 install hf_transfer

HF_HUB_ENABLE_HF_TRANSFER = 1 huggingface - cli download TheBloke/Norocetacean-20B-10k-GGUF norocetacean-20b-10k.Q4_K_M.gguf --local - dir . --local - dir - use - symlinks False

💻 Example llama.cpp Command

Make sure you are using llama.cpp from commit d0cee0d or later.

./main -ngl 35 -m norocetacean-20b-10k.Q4_K_M.gguf --color -c 10240 --temp 0.7 --repeat_penalty 1.1 -n -1 -p "Below is an instruction that describe