Gemma 3 12b It FP8 Dynamic

🚀 gemma-3-12b-it-FP8-Dynamic

A quantized version of the gemma-3-12b-it model, optimized for efficient inference with vLLM.

🚀 Quick Start

This model can be deployed efficiently using the vLLM backend. Check the "Deployment" section below for a detailed example.

✨ Features

• Multimodal Input: Supports both vision and text inputs, enabling a wide range of applications.
• Quantization Optimization: Uses FP8 quantization for both weights and activations, reducing memory usage and accelerating inference.
• Efficient Inference: Compatible with vLLM, which provides high-throughput serving and OpenAI-compatible APIs.

📦 Installation

There is no specific installation step provided for this model. However, you need to have the necessary dependencies installed, such as vLLM, transformers, and llm-compressor. You can install them using pip:

pip install vllm transformers llmcompressor

💻 Usage Examples

Basic Usage

This model can be deployed efficiently using the vLLM backend, as shown in the example below.

from vllm import LLM, SamplingParams
from vllm.assets.image import ImageAsset
from transformers import AutoProcessor

# Define model name once
model_name = "RedHatAI/gemma-3-12b-it-FP8-dynamic"

# Load image and processor
image = ImageAsset("cherry_blossom").pil_image.convert("RGB")
processor = AutoProcessor.from_pretrained(model_name, trust_remote_code=True)

# Build multimodal prompt
chat = [
    {"role": "user", "content": [{"type": "image"}, {"type": "text", "text": "What is the content of this image?"}]},
    {"role": "assistant", "content": []}
]
prompt = processor.apply_chat_template(chat, add_generation_prompt=True)

# Initialize model
llm = LLM(model=model_name, trust_remote_code=True)

# Run inference
inputs = {"prompt": prompt, "multi_modal_data": {"image": [image]}}
outputs = llm.generate(inputs, SamplingParams(temperature=0.2, max_tokens=64))

# Display result
print("RESPONSE:", outputs[0].outputs[0].text)

vLLM also supports OpenAI-compatible serving. See the documentation for more details.

📚 Documentation

Model Overview

• Model Architecture: gemma-3-12b-it
  • Input: Vision-Text
  • Output: Text
• Model Optimizations:
  • Weight quantization: FP8
  • Activation quantization: FP8
• Release Date: 2/24/2025
• Version: 1.0
• Model Developers: Neural Magic

This is a quantized version of google/gemma-3-12b-it.

Model Optimizations

This model was obtained by quantizing the weights of google/gemma-3-12b-it to FP8 data type, ready for inference with vLLM >= 0.5.2.

Deployment

This model can be deployed using the vLLM backend. See the "Usage Examples" section for a code example.

Creation

This model was created with llm-compressor by running the code snippet below as part a multimodal announcement blog.

import requests
import torch
from PIL import Image
from transformers import AutoProcessor, Gemma3ForConditionalGeneration
from llmcompressor.transformers import oneshot
from llmcompressor.modifiers.quantization import QuantizationModifier

# Load model.
model_id = "google/gemma-3-12b-it"
model = Gemma3ForConditionalGeneration.from_pretrained(
    model_id, device_map="auto", torch_dtype="auto"
)
processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)

# Recipe
recipe = [
    QuantizationModifier(
        targets="Linear",
        scheme="FP8_DYNAMIC",
        sequential_targets=["Gemma3DecoderLayer"],
        ignore=["re:.*lm_head", "re:vision_tower.*", "re:multi_modal_projector.*"],
    ),
]

SAVE_DIR=f"{model_id.split('/')[1]}-FP8-Dynamic"

# Perform oneshot
oneshot(
    model=model,
    recipe=recipe,
    trust_remote_code_model=True,
    output_dir=SAVE_DIR
)

Evaluation

The model was evaluated using lm_evaluation_harness for OpenLLM v1 text benchmark. The evaluations were conducted using the following commands:

lm_eval \
  --model vllm \
  --model_args pretrained="<model_name>",dtype=auto,add_bos_token=True,max_model_len=4096,tensor_parallel_size=<n>,gpu_memory_utilization=0.8,enable_chunked_prefill=True,trust_remote_code=True,enforce_eager=True \
  --tasks openllm \
  --batch_size auto

Accuracy

📄 License

This model is licensed under the Apache 2.0 license.