đ gemma-3-27b-it-quantized.w4a16
This is a quantized version of google/gemma-3-27b-it, optimized for efficient inference.
đ Quick Start
This model is a quantized version of google/gemma-3-27b-it, which can be efficiently deployed with vLLM. Below are the details of model deployment, creation, and evaluation.
⨠Features
- Model Architecture: Based on google/gemma-3-27b-it, supporting vision-text input and text output.
- Model Optimizations: Utilizes INT4 weight quantization and FP16 activation quantization.
- Release Date: 6/4/2025
- Version: 1.0
- Model Developers: RedHatAI
Property |
Details |
Model Type |
Quantized version of google/gemma-3-27b-it |
Training Data |
Not specified |
đĻ Installation
This model can be deployed efficiently using the vLLM backend.
đģ Usage Examples
Basic Usage
from vllm import LLM, SamplingParams
from vllm.assets.image import ImageAsset
from transformers import AutoProcessor
model_name = "RedHatAI/gemma-3-27b-it-quantized.w4a16"
image = ImageAsset("cherry_blossom").pil_image.convert("RGB")
processor = AutoProcessor.from_pretrained(model_name, trust_remote_code=True)
chat = [
{"role": "user", "content": [{"type": "image"}, {"type": "text", "text": "What is the content of this image?"}]},
{"role": "assistant", "content": []}
]
prompt = processor.apply_chat_template(chat, add_generation_prompt=True)
llm = LLM(model=model_name, trust_remote_code=True)
inputs = {"prompt": prompt, "multi_modal_data": {"image": [image]}}
outputs = llm.generate(inputs, SamplingParams(temperature=0.2, max_tokens=64))
print("RESPONSE:", outputs[0].outputs[0].text)
vLLM also supports OpenAI-compatible serving. See the documentation for more details.
đ Documentation
Creation
This model was created with llm-compressor by running the code snippet below:
Model Creation Code
import base64
from io import BytesIO
import torch
from datasets import load_dataset
from transformers import AutoProcessor, Gemma3ForConditionalGeneration
from llmcompressor.modifiers.quantization import GPTQModifier
from llmcompressor.transformers import oneshot
model_id = "google/gemma-3-27b-it"
model = Gemma3ForConditionalGeneration.from_pretrained(
model_id,
device_map="auto",
torch_dtype="auto",
)
processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)
DATASET_ID = "neuralmagic/calibration"
DATASET_SPLIT = {"LLM": "train[:1024]"}
NUM_CALIBRATION_SAMPLES = 1024
MAX_SEQUENCE_LENGTH = 2048
ds = load_dataset(DATASET_ID, split=DATASET_SPLIT)
ds = ds.shuffle(seed=42)
dampening_frac=0.07
def data_collator(batch):
assert len(batch) == 1, "Only batch size of 1 is supported for calibration"
item = batch[0]
collated = {}
import torch
for key, value in item.items():
if isinstance(value, torch.Tensor):
collated[key] = value.unsqueeze(0)
elif isinstance(value, list) and isinstance(value[0][0], int):
collated[key] = torch.tensor(value)
elif isinstance(value, list) and isinstance(value[0][0], float):
collated[key] = torch.tensor(value)
elif isinstance(value, list) and isinstance(value[0][0], torch.Tensor):
collated[key] = torch.stack(value)
elif isinstance(value, torch.Tensor):
collated[key] = value
else:
print(f"[WARN] Unrecognized type in collator for key={key}, type={type(value)}")
return collated
recipe = [
GPTQModifier(
targets="Linear",
ignore=["re:.*lm_head.*", "re:.*embed_tokens.*", "re:vision_tower.*", "re:multi_modal_projector.*"],
sequential_update=True,
sequential_targets=["Gemma3DecoderLayer"],
dampening_frac=dampening_frac,
)
]
SAVE_DIR=f"{model_id.split('/')[1]}-quantized.w4a16"
oneshot(
model=model,
tokenizer=model_id,
dataset=ds,
recipe=recipe,
max_seq_length=MAX_SEQUENCE_LENGTH,
num_calibration_samples=NUM_CALIBRATION_SAMPLES,
trust_remote_code_model=True,
data_collator=data_collator,
output_dir=SAVE_DIR
)
Evaluation
The model was evaluated using lm_evaluation_harness for OpenLLM v1 text benchmark. The evaluations were conducted using the following commands:
Evaluation Commands
OpenLLM v1
lm_eval \
--model vllm \
--model_args pretrained="<model_name>",dtype=auto,add_bos_token=True,max_model_len=4096,tensor_parallel_size=<n>,gpu_memory_utilization=0.8,enable_chunked_prefill=True,trust_remote_code=True,enforce_eager=True \
--tasks openllm \
--batch_size auto
Accuracy
Category |
Metric |
google/gemma-3-27b-it |
RedHatAI/gemma-3-27b-it-quantized.w8a8 |
Recovery (%) |
OpenLLM V1 |
ARC Challenge |
72.53% |
72.35% |
99.76% |
|
GSM8K |
92.12% |
91.66% |
99.51% |
|
Hellaswag |
85.78% |
84.97% |
99.06% |
|
MMLU |
77.53% |
76.77% |
99.02% |
|
Truthfulqa (mc2) |
62.20% |
62.57% |
100.59% |
|
Winogrande |
79.40% |
79.79% |
100.50% |
|
Average Score |
78.26% |
78.02% |
99.70% |
Vision Evals |
MMMU (val) |
50.89% |
51.78% |
101.75% |
|
ChartQA |
72.16% |
72.20% |
100.06% |
|
Average Score |
61.53% |
61.99% |
100.90% |
đ License
The model is under the gemma license.