# 4-bit quantization inference

Kimi-K2-Instruct-4bit is a 4-bit quantized model converted from moonshotai/Kimi-K2-Instruct, suitable for the MLX framework.

The 4-bit quantized version of Qwen3-30B-A3B, suitable for efficient inference on the MLX framework

The 4-bit AWQ quantized version of the DeepSeek-R1-0528 671B model, suitable for use on high-end GPU nodes

A 4-bit quantized version of the Gemma 3 12B model, suitable for the MLX framework and supporting efficient text generation tasks.

This is a model that uses the GPTQ method to perform 4-bit quantization on GLM-4-32B-0414, suitable for consumer-grade hardware.

Gemma 2 27B IT is a 4-bit large language model based on AutoAWQ quantization, suitable for dialogue and instruction-following tasks.

The AWQ 4-bit quantization version of QwQ-32B-Preview significantly reduces memory usage and computational requirements, making it suitable for hardware deployment with limited resources.

The 4-bit quantized version of Mistral-7B-Instruct-v0.3, which optimizes inference performance through the GPTQ method while maintaining high accuracy

A medical Q&A model fine-tuned on the MedQuAD dataset based on Llama-2-7b-chat

The 4-bit quantized version of Falcon-7B-Instruct, quantized using the AutoGPTQ tool, suitable for efficient inference in resource-constrained environments.