Cogito 14b Gptq Q4

🚀 Transformers Quantization Project

This project focuses on quantizing the Qwen/Qwen2.5-14B model using GPTQModel for text generation tasks.

🚀 Quick Start

The model is quantized with GPTQModel 4.0.0 dev. The following steps and code demonstrate how to perform the quantization.

💻 Usage Examples

Basic Usage

# 1. grab 512 English + 512 Spanish documents
from datasets import load_dataset, concatenate_datasets
from gptqmodel import GPTQModel, QuantizeConfig
from random import shuffle, seed

seed(0)

en_ds = load_dataset("allenai/c4", data_files="en/c4-train.00001-of-01024.json.gz", split="train") \
          .shuffle(seed=0).select(range(512))
es_ds = load_dataset("allenai/c4", data_files="multilingual/c4-es.tfrecord-00001-of-02048.json.gz", split="train") \
          .shuffle(seed=0).select(range(512))

calib_texts = [x["text"] for x in concatenate_datasets([en_ds, es_ds])]
shuffle(calib_texts)

# 2. quantise
model_id  = "deepcogito/cogito-v1-preview-qwen-14B"
quant_dir = "cogito-14b-gptq-q4"

qconf = QuantizeConfig(bits=4, group_size=128)
model = GPTQModel.load(model_id, qconf)

# model.quantize(calib_texts, batch_size=2)
model.quantize(calib_texts, batch_size=1)
model.save(quant_dir)

Advanced Usage

For the calibration dataset, 50% Spanish and 50% English texts are included as the tasks mainly involve these two languages.

💡 Usage Tip

Note that the author tried the v2 version but found it had a much higher loss compared with v1, so the v1 version is used.

📄 License

This project is licensed under the Apache-2.0 license.