Gemma 3 4b It Quantized W4A16

🚀 Gemma 3 Quantized Models

This repository offers W4A16 quantized versions of Google's Gemma 3 instruction - tuned models. These models are more deployable on consumer hardware without sacrificing much performance.

🚀 Quick Start

This repository contains quantized versions of Google's Gemma 3 instruction - tuned models, which are more suitable for deployment on consumer hardware while maintaining good performance.

✨ Features

• Quantized Models: The repository provides W4A16 quantized versions of Gemma 3 models, including abhishekchohan/gemma - 3 - 27b - it - quantized - W4A16, abhishekchohan/gemma - 3 - 12b - it - quantized - W4A16, and abhishekchohan/gemma - 3 - 4b - it - quantized - W4A16.
• Reduced Memory Requirements: Through W4A16 quantization, the models have significantly reduced memory requirements.

📦 Installation

To use these models, you need to access Gemma on Hugging Face.

⚠️ Important Note

To access Gemma on Hugging Face, you're required to review and agree to Google's usage license. To do this, please ensure you're logged in to Hugging Face and click the "Acknowledge license" button. Requests are processed immediately.

💻 Usage Examples

Basic Usage

vllm serve abhishekchohan/gemma-3-{size}-it-quantized-W4A16 --chat-template templates/chat_template.jinja --enable-auto-tool-choice --tool-call-parser gemma --tool-parser-plugin tools/tool_parser.py

📚 Documentation

Repository Structure

gemma-3-{size}-it-quantized-W4A16/
├── README.md
├── templates/
│   └── chat_template.jinja
├── tools/
│   └── tool_parser.py
└── [model files]

Quantization Details

These models use W4A16 quantization via LLM Compressor:

• Weights quantized to 4 - bit precision
• Activations use 16 - bit precision
• Significantly reduced memory requirements

📄 License

These models are subject to the Gemma license. Users must acknowledge and accept the license terms before using the models.

📚 Citation

@article{gemma_2025,
    title={Gemma 3},
    url={https://goo.gle/Gemma3Report},
    publisher={Kaggle},
    author={Gemma Team},
    year={2025}
}