Gpt4 X Alpaca 13b Native 4bit 128g

🚀 GPT4-X Alpaca Model Project

This project provides quantized models of GPT4-X Alpaca and related running commands.

🚀 Quick Start

There are currently two different models, one generated in the Triton branch and the other in Cuda. It is recommended to use the Cuda model for now unless the Triton branch becomes widely adopted.

✨ Features

• Model Update: As of April 1st, ggml has been added for the Cuda model.
• Dataset Availability: The instruction dataset can be found at https://github.com/teknium1/GPTeacher.

📦 Installation

Cuda Model

You can run the following command to use the Cuda model:

CUDA_VISIBLE_DEVICES=0 python llama.py ./models/chavinlo-gpt4-x-alpaca --wbits 4 --true-sequential --groupsize 128 --save gpt-x-alpaca-13b-native-4bit-128g-cuda.pt

Previous Model

This model was quantized on the GPTQ - for - LLaMa commit 5955e9c67d9bfe8a8144ffbe853c2769f1e87cdd. It is a GPTQ 4 - bit quantization of https://huggingface.co/chavinlo/gpt4-x-alpaca.

Note that this model was quantized using this branch of GPTQ - for - LLaMa: https://github.com/qwopqwop200/GPTQ-for-LLaMa/tree/triton. Currently, it seems to be incompatible with Oobabooga.

You can run the following command to use the previous model:

CUDA_VISIBLE_DEVICES=0 python llama.py ./models/chavinlo-gpt4-x-alpaca --wbits 4 --true-sequential --act-order --groupsize 128 --save gpt-x-alpaca-13b-native-4bit-128g.pt

⚠️ Important Note

The model quantized using the specified branch of GPTQ - for - LLaMa is currently incompatible with Oobabooga. Please stay tuned for further updates.