Llama2 13B Tiefighter GPTQ

🚀 Llama2 13B Tiefighter - GPTQ

This repository provides GPTQ model files for KoboldAI's Llama2 13B Tiefighter, offering multiple quantisation options for different hardware and requirements.

📚 Documentation

Model Information

Model Description

This repo contains GPTQ model files for KoboldAI's Llama2 13B Tiefighter. Multiple GPTQ parameter permutations are provided; see Provided Files below for details of the options provided, their parameters, and the software used to create them.

Repositories available

• AWQ model(s) for GPU inference.
• GPTQ models for GPU inference, with multiple quantisation parameter options.
• 2, 3, 4, 5, 6 and 8 - bit GGUF models for CPU+GPU inference
• KoboldAI's original unquantised fp16 model in pytorch format, for GPU inference and for further conversions

Prompt template: Alpaca - Tiefighter

### Instruction: 
{prompt}
### Response:

Known compatible clients / servers

These GPTQs are known to work in the following inference servers/webuis:

• [text - generation - webui](https://github.com/oobabooga/text - generation - webui)
• KobaldAI United
• [LoLLMS Web UI](https://github.com/ParisNeo/lollms - webui)
• [Hugging Face Text Generation Inference (TGI)](https://github.com/huggingface/text - generation - inference)

Provided files, and GPTQ parameters

Multiple quantisation parameters are provided, to allow you to choose the best one for your hardware and requirements. Each separate quant is in a different branch. See below for instructions on fetching from different branches. Most GPTQ files are made with AutoGPTQ. Mistral models are currently made with Transformers.

How to download, including from branches

In text - generation - webui

To download from the main branch, enter TheBloke/LLaMA2-13B-Tiefighter-GPTQ in the "Download model" box. To download from another branch, add :branchname to the end of the download name, eg TheBloke/LLaMA2-13B-Tiefighter-GPTQ:gptq-4bit-32g-actorder_True

From the command line

I recommend using the huggingface - hub Python library:

pip3 install huggingface - hub

To download the main branch to a folder called LLaMA2-13B-Tiefighter-GPTQ:

mkdir LLaMA2-13B-Tiefighter-GPTQ
huggingface - cli download TheBloke/LLaMA2-13B-Tiefighter-GPTQ --local - dir LLaMA2-13B-Tiefighter-GPTQ --local - dir - use - symlinks False

To download from a different branch, add the --revision parameter:

mkdir LLaMA2-13B-Tiefighter-GPTQ
huggingface - cli download TheBloke/LLaMA2-13B-Tiefighter-GPTQ --revision gptq-4bit-32g-actorder_True --local - dir LLaMA2-13B-Tiefighter-GPTQ --local - dir - use - symlinks False

pip3 install hf_transfer

mkdir LLaMA2-13B-Tiefighter-GPTQ
HF_HUB_ENABLE_HF_TRANSFER = 1 huggingface - cli download TheBloke/LLaMA2-13B-Tiefighter-GPTQ --local - dir LLaMA2-13B-Tiefighter-GPTQ --local - dir - use - symlinks False

With git (not recommended)

To clone a specific branch with git, use a command like this:

git clone --single - branch --branch gptq-4bit-32g-actorder_True https://huggingface.co/TheBloke/LLaMA2-13B-Tiefighter-GPTQ

Note that using Git with HF repos is strongly discouraged. It will be much slower than using huggingface - hub, and will use twice as much disk space as it has to store the model files twice (it stores every byte both in the intended target folder, and again in the .git folder as a blob.)

How to easily download and use this model in [text - generation - webui](https://github.com/oobabooga/text - generation - webui)

Please make sure you're using the latest version of [text - generation - webui](https://github.com/oobabooga/text - generation - webui). It is strongly recommended to use the text - generation - webui one - click - installers unless you're sure you know how to make a manual install.

1. Click the Model tab.
2. Under Download custom model or LoRA, enter TheBloke/LLaMA2-13B-Tiefighter-GPTQ.
   • To download from a specific branch, enter for example TheBloke/LLaMA2-13B-Tiefighter-GPTQ:gptq-4bit-32g-actorder_True
   • see Provided Files above for the list of branches for each option.
3. Click Download.
4. The model will start downloading. Once it's finished it will say "Done".
5. In the top left, click the refresh icon next to Model.
6. In the Model dropdown, choose the model you just downloaded: LLaMA2-13B-Tiefighter-GPTQ
7. The model will automatically load, and is now ready for use!
8. If you want any custom settings, set them and then click Save settings for this model followed by Reload the Model in the top right.
   • Note that you do not need to and should not set manual GPTQ parameters any more. These are set automatically from the file quantize_config.json.
9. Once you're ready, click the Text Generation tab and enter a prompt to get started!

Serving this model from Text Generation Inference (TGI)

It's recommended to use TGI version 1.1.0 or later. The official Docker container is: ghcr.io/huggingface/text - generation - inference:1.1.0

Example Docker parameters:

--model - id TheBloke/LLaMA2-13B-Tiefighter-GPTQ --port 3000 --quantize gptq --max - input - length 3696 --max - total - tokens 4096 --max - batch - prefill - tokens 4096

Example Python code for interfacing with TGI (requires huggingface - hub 0.17.0 or later):

pip3 install huggingface - hub

from huggingface_hub import InferenceClient

endpoint_url = "https://your - endpoint - url - here"

prompt = "Tell me about AI"