Gemma 3 27b It Quantized W4A16

🚀 Gemma 3 Quantized Models

This repository offers W4A16 quantized versions of Google's Gemma 3 instruction - tuned models. These versions are more accessible for deployment on consumer hardware while still maintaining good performance.

🚀 Quick Start

To use the quantized models with vLLM, you can run the following command:

vllm serve abhishekchohan/gemma-3-{size}-it-quantized-W4A16 --chat-template templates/chat_template.jinja --enable-auto-tool-choice --tool-call-parser gemma --tool-parser-plugin tools/tool_parser.py

✨ Features

• Access Gemma on Hugging Face: To access Gemma on Hugging Face, you're required to review and agree to Google's usage license. To do this, please ensure you're logged in to Hugging Face and click below. Requests are processed immediately. [Acknowledge license]
• The repository contains multiple quantized models, including:
  • abhishekchohan/gemma-3-27b-it-quantized-W4A16
  • abhishekchohan/gemma-3-12b-it-quantized-W4A16
  • abhishekchohan/gemma-3-4b-it-quantized-W4A16
• The models use W4A16 quantization via LLM Compressor, which significantly reduces memory requirements while maintaining performance.

📦 Installation

There is no specific installation content provided in the original README. If you want to use the models, you can follow the usage command mentioned above.

💻 Usage Examples

Basic Usage

vllm serve abhishekchohan/gemma-3-{size}-it-quantized-W4A16 --chat-template templates/chat_template.jinja --enable-auto-tool-choice --tool-call-parser gemma --tool-parser-plugin tools/tool_parser.py

📚 Documentation

Repository Structure

gemma-3-{size}-it-quantized-W4A16/
├── README.md
├── templates/
│   └── chat_template.jinja
├── tools/
│   └── tool_parser.py
└── [model files]

Quantization Details

These models use W4A16 quantization via LLM Compressor:

• Weights quantized to 4 - bit precision
• Activations use 16 - bit precision
• Significantly reduced memory requirements

🔧 Technical Details

The models use the W4A16 quantization method through LLM Compressor. By quantizing the weights to 4 - bit precision and using 16 - bit precision for activations, the memory requirements are greatly reduced, which allows the models to be deployed on consumer hardware more easily.

📄 License

These models are subject to the Gemma license. Users must acknowledge and accept the license terms before using the models.

📚 Citation

@article{gemma_2025,
    title={Gemma 3},
    url={https://goo.gle/Gemma3Report},
    publisher={Kaggle},
    author={Gemma Team},
    year={2025}
}