Deploy the Gemma 3-27B IT quantitative model for free. It supports visual and text input and output, and features efficient inference.

Gemma 3 27b It Quantized.w4a16

Developed by RedHatAI

This is a quantized version of google/gemma-3-27b-it, supporting visual-text input and text output. Optimized through weight quantization and activation quantization, it enables efficient inference with vLLM.

Image-to-Text

Transformers

#Multimodal understanding #Efficient inference #INT4 quantization

Downloads 302

Release Time : 6/4/2025

Model Overview

A quantized model based on google/gemma-3-27b-it, supporting multimodal input and text generation, suitable for scenarios requiring efficient inference.

Model Features

Efficient quantization

Adopts INT4 weight quantization and FP16 activation quantization, significantly reducing model size and inference resource requirements.

Multimodal support

Supports visual-text input and can handle complex tasks combining images and text.

Efficient inference

After optimization, it can be efficiently deployed and inferred through vLLM, improving processing speed.

Model Capabilities

Multimodal understanding

Text generation

Image content analysis

Use Cases

Content understanding and generation

Image content description

Analyze image content and generate descriptive text

Accurately identify the main elements and scenes in the image

Multimodal dialogue

Conduct intelligent dialogue by combining image and text input

Provide coherent answers related to the image content

🚀 gemma-3-27b-it-quantized.w4a16

This is a quantized version of google/gemma-3-27b-it, optimized for efficient inference.

🚀 Quick Start

This model is a quantized version of google/gemma-3-27b-it, which can be efficiently deployed with vLLM. Below are the details of model deployment, creation, and evaluation.

✨ Features

Model Architecture: Based on google/gemma-3-27b-it, supporting vision-text input and text output.
Model Optimizations: Utilizes INT4 weight quantization and FP16 activation quantization.
Release Date: 6/4/2025
Version: 1.0
Model Developers: RedHatAI

Property	Details
Model Type	Quantized version of google/gemma-3-27b-it
Training Data	Not specified

📦 Installation

This model can be deployed efficiently using the vLLM backend.

💻 Usage Examples

Basic Usage

from vllm import LLM, SamplingParams
from vllm.assets.image import ImageAsset
from transformers import AutoProcessor

# Define model name once
model_name = "RedHatAI/gemma-3-27b-it-quantized.w4a16"

# Load image and processor
image = ImageAsset("cherry_blossom").pil_image.convert("RGB")
processor = AutoProcessor.from_pretrained(model_name, trust_remote_code=True)

# Build multimodal prompt
chat = [
    {"role": "user", "content": [{"type": "image"}, {"type": "text", "text": "What is the content of this image?"}]},
    {"role": "assistant", "content": []}
]
prompt = processor.apply_chat_template(chat, add_generation_prompt=True)

# Initialize model
llm = LLM(model=model_name, trust_remote_code=True)

# Run inference
inputs = {"prompt": prompt, "multi_modal_data": {"image": [image]}}
outputs = llm.generate(inputs, SamplingParams(temperature=0.2, max_tokens=64))

# Display result
print("RESPONSE:", outputs[0].outputs[0].text)

vLLM also supports OpenAI-compatible serving. See the documentation for more details.

📚 Documentation

Creation

This model was created with llm-compressor by running the code snippet below:

Model Creation Code

import base64
from io import BytesIO
import torch
from datasets import load_dataset
from transformers import AutoProcessor, Gemma3ForConditionalGeneration
from llmcompressor.modifiers.quantization import GPTQModifier
from llmcompressor.transformers import oneshot


# Load model.
model_id = "google/gemma-3-27b-it"
model = Gemma3ForConditionalGeneration.from_pretrained(
    model_id,
    device_map="auto",
    torch_dtype="auto",
)
processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)

# Oneshot arguments
DATASET_ID = "neuralmagic/calibration"
DATASET_SPLIT = {"LLM": "train[:1024]"}
NUM_CALIBRATION_SAMPLES = 1024
MAX_SEQUENCE_LENGTH = 2048

# Load dataset and preprocess.
ds = load_dataset(DATASET_ID, split=DATASET_SPLIT)
ds = ds.shuffle(seed=42)

dampening_frac=0.07

def data_collator(batch):
    assert len(batch) == 1, "Only batch size of 1 is supported for calibration"
    item = batch[0]
    collated = {}
    import torch


    for key, value in item.items():
        if isinstance(value, torch.Tensor):
            collated[key] = value.unsqueeze(0)
        elif isinstance(value, list) and isinstance(value[0][0], int):
            # Handle tokenized inputs like input_ids, attention_mask
            collated[key] = torch.tensor(value)
        elif isinstance(value, list) and isinstance(value[0][0], float):
            # Handle possible float sequences
            collated[key] = torch.tensor(value)
        elif isinstance(value, list) and isinstance(value[0][0], torch.Tensor):
            # Handle batched image data (e.g., pixel_values as [C, H, W])
            collated[key] = torch.stack(value)  # -> [1, C, H, W]
        elif isinstance(value, torch.Tensor):
            collated[key] = value
        else:
            print(f"[WARN] Unrecognized type in collator for key={key}, type={type(value)}")
    
    return collated
   


# Recipe
recipe = [
    GPTQModifier(
        targets="Linear",
        ignore=["re:.*lm_head.*", "re:.*embed_tokens.*", "re:vision_tower.*", "re:multi_modal_projector.*"],
        sequential_update=True,
        sequential_targets=["Gemma3DecoderLayer"],
        dampening_frac=dampening_frac,
    )
]

SAVE_DIR=f"{model_id.split('/')[1]}-quantized.w4a16"

# Perform oneshot
oneshot(
    model=model,
    tokenizer=model_id,
    dataset=ds,
    recipe=recipe,
    max_seq_length=MAX_SEQUENCE_LENGTH,
    num_calibration_samples=NUM_CALIBRATION_SAMPLES,
    trust_remote_code_model=True,
    data_collator=data_collator,
    output_dir=SAVE_DIR
)

Evaluation

The model was evaluated using lm_evaluation_harness for OpenLLM v1 text benchmark. The evaluations were conducted using the following commands:

Evaluation Commands

OpenLLM v1

lm_eval \
  --model vllm \
  --model_args pretrained="<model_name>",dtype=auto,add_bos_token=True,max_model_len=4096,tensor_parallel_size=<n>,gpu_memory_utilization=0.8,enable_chunked_prefill=True,trust_remote_code=True,enforce_eager=True \
  --tasks openllm \
  --batch_size auto

Accuracy

Category	Metric	google/gemma-3-27b-it	RedHatAI/gemma-3-27b-it-quantized.w8a8	Recovery (%)
OpenLLM V1	ARC Challenge	72.53%	72.35%	99.76%
	GSM8K	92.12%	91.66%	99.51%
	Hellaswag	85.78%	84.97%	99.06%
	MMLU	77.53%	76.77%	99.02%
	Truthfulqa (mc2)	62.20%	62.57%	100.59%
	Winogrande	79.40%	79.79%	100.50%
	Average Score	78.26%	78.02%	99.70%
Vision Evals	MMMU (val)	50.89%	51.78%	101.75%
	ChartQA	72.16%	72.20%	100.06%
	Average Score	61.53%	61.99%	100.90%

📄 License

The model is under the gemma license.

Featured Recommended AI Models

Empowering the Future, Your AI Solution Knowledge Base

English 简体中文繁體中文にほんご