Storytime 13B GPTQ

🚀 Storytime 13B - GPTQ

This repository contains GPTQ model files for Charles Goddard's Storytime 13B, offering multiple quantisation parameter options to suit different hardware and requirements.

✨ Features

• Multiple Quantisation Options: Different GPTQ parameter permutations are provided to meet various hardware and performance needs.
• Diverse Repositories: Available in AWQ, GPTQ, GGUF formats, as well as the original unquantised fp16 model.
• Alpaca Prompt Template: Uses the Alpaca prompt template for easy interaction.

📦 Installation

In text - generation - webui

To download from the main branch, enter TheBloke/storytime-13B-GPTQ in the "Download model" box. To download from another branch, add :branchname to the end of the download name, e.g., TheBloke/storytime-13B-GPTQ:gptq-4-32g-actorder_True

From the command line

I recommend using the huggingface - hub Python library:

pip3 install huggingface - hub

To download the main branch to a folder called storytime-13B-GPTQ:

mkdir storytime-13B-GPTQ
huggingface-cli download TheBloke/storytime-13B-GPTQ --local-dir storytime-13B-GPTQ --local-dir-use-symlinks False

To download from a different branch, add the --revision parameter:

mkdir storytime-13B-GPTQ
huggingface-cli download TheBloke/storytime-13B-GPTQ --revision gptq-4-32g-actorder_True --local-dir storytime-13B-GPTQ --local-dir-use-symlinks False

With git (not recommended)

To clone a specific branch with git, use a command like this:

git clone --single-branch --branch gptq-4-32g-actorder_True https://huggingface.co/TheBloke/storytime-13B-GPTQ

💻 Usage Examples

How to easily download and use this model in [text - generation - webui](https://github.com/oobabooga/text - generation - webui)

1. Click the Model tab.
2. Under Download custom model or LoRA, enter TheBloke/storytime-13B-GPTQ.
   • To download from a specific branch, enter for example TheBloke/storytime-13B-GPTQ:gptq-4-32g-actorder_True
   • see Provided Files below for the list of branches for each option.
3. Click Download.
4. The model will start downloading. Once it's finished it will say "Done".
5. In the top left, click the refresh icon next to Model.
6. In the Model dropdown, choose the model you just downloaded: storytime-13B-GPTQ
7. The model will automatically load, and is now ready for use!
8. If you want any custom settings, set them and then click Save settings for this model followed by Reload the Model in the top right.
   • Note that you do not need to and should not set manual GPTQ parameters any more. These are set automatically from the file quantize_config.json.
9. Once you're ready, click the Text Generation tab and enter a prompt to get started!

How to use this GPTQ model from Python code

Install the necessary packages

Requires: Transformers 4.33.0 or later, Optimum 1.12.0 or later, and AutoGPTQ 0.4.2 or later.

pip3 install transformers optimum
pip3 install auto - gptq --extra - index - url https://huggingface.github.io/autogptq - index/whl/cu118/  # Use cu117 if on CUDA 11.7

If you have problems installing AutoGPTQ using the pre - built wheels, install it from source instead:

pip3 uninstall - y auto - gptq
git clone https://github.com/PanQiWei/AutoGPTQ
cd AutoGPTQ
git checkout v0.4.2
pip3 install .

You can then use the following code

from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline

model_name_or_path = "TheBloke/storytime-13B-GPTQ"
# To use a different branch, change revision
# For example: revision="gptq-4-32g-actorder_True"
model = AutoModelForCausalLM.from_pretrained(model_name_or_path,
                                             device_map="auto",
                                             trust_remote_code=False,
                                             revision="main")

tokenizer = AutoTokenizer.from_pretrained(model_name_or_path, use_fast=True)

# Create a text generation pipeline
generate_text = pipeline('text-generation',
                         model=model,
                         tokenizer=tokenizer,
                         device_map="auto")

# Generate text
prompt = "Once upon a time"
output = generate_text(prompt, max_length=200, num_return_sequences=1)
print(output[0]['generated_text'])

📚 Documentation

Model Information

• Model creator: Charles Goddard
• Original model: Storytime 13B

Repositories available

• AWQ model(s) for GPU inference.
• GPTQ models for GPU inference, with multiple quantisation parameter options.
• 2, 3, 4, 5, 6 and 8 - bit GGUF models for CPU+GPU inference
• Charles Goddard's original unquantised fp16 model in pytorch format, for GPU inference and for further conversions

Prompt template: Alpaca

Below is an instruction that describes a task. Write a response that appropriately completes the request.

### Instruction:
{prompt}

### Response:

Provided files, and GPTQ parameters

Multiple quantisation parameters are provided, to allow you to choose the best one for your hardware and requirements. Each separate quant is in a different branch. See below for instructions on fetching from different branches. All recent GPTQ files are made with AutoGPTQ, and all files in non - main branches are made with AutoGPTQ. Files in the main branch which were uploaded before August 2023 were made with GPTQ - for - LLaMa.

📄 License

The model is under the Llama 2 license.