Gemma 3 27b It Int4 Gguf

🚀 Gemma 3 27B Instruction-tuned INT4

This is a QAT INT4 Flax checkpoint (from Kaggle) converted to GGUF format for easy use. You can find the conversion script at GitHub. Note that this is not the same as the official QAT INT4 GGUFs released here. Below is the original Model card from Google Gemma 3 27B IT.

🚀 Quick Start

Access Gemma on Hugging Face

To access Gemma on Hugging Face, you’re required to review and agree to Google’s usage license. To do this, please ensure you’re logged in to Hugging Face and click below. Requests are processed immediately. [Acknowledge license]

Installation

First, install the Transformers library. Gemma 3 is supported starting from transformers 4.50.0.

$ pip install -U transformers

Usage Examples

Basic Usage

You can initialize the model and processor for inference with pipeline as follows.

from transformers import pipeline
import torch

pipe = pipeline(
    "image-text-to-text",
    model="google/gemma-3-27b-it",
    device="cuda",
    torch_dtype=torch.bfloat16
)

With instruction-tuned models, you need to use chat templates to process our inputs first. Then, you can pass it to the pipeline.

messages = [
    {
        "role": "system",
        "content": [{"type": "text", "text": "You are a helpful assistant."}]
    },
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    }
]

output = pipe(text=messages, max_new_tokens=200)
print(output[0]["generated_text"][-1]["content"])
# Okay, let's take a look! 
# Based on the image, the animal on the candy is a **turtle**. 
# You can see the shell shape and the head and legs.

Advanced Usage

Running the model on a single/multi GPU

# pip install accelerate

from transformers import AutoProcessor, Gemma3ForConditionalGeneration
from PIL import Image
import requests
import torch

model_id = "google/gemma-3-27b-it"

model = Gemma3ForConditionalGeneration.from_pretrained(
    model_id, device_map="auto"
).eval()

processor = AutoProcessor.from_pretrained(model_id)

messages = [
    {
        "role": "system",
        "content": [{"type": "text", "text": "You are a helpful assistant."}]
    },
    {
        "role": "user",
        "content": [
            {"type": "image", "image": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg"},
            {"type": "text", "text": "Describe this image in detail."}
        ]
    }
]

inputs = processor.apply_chat_template(
    messages, add_generation_prompt=True, tokenize=True,
    return_dict=True, return_tensors="pt"
).to(model.device, dtype=torch.bfloat16)

input_len = inputs["input_ids"].shape[-1]

with torch.inference_mode():
    generation = model.generate(**inputs, max_new_tokens=100, do_sample=False)
    generation = generation[0][input_len:]

decoded = processor.decode(generation, skip_special_tokens=True)
print(decoded)

# **Overall Impression:** The image is a close-up shot of a vibrant garden scene, 
# focusing on a cluster of pink cosmos flowers and a busy bumblebee. 
# It has a slightly soft, natural feel, likely captured in daylight.

✨ Features

Model Information

• Summary: Gemma is a family of lightweight, state-of-the-art open models from Google, built from the same research and technology used to create the Gemini models. Gemma 3 models are multimodal, handling text and image input and generating text output, with open weights for both pre-trained variants and instruction-tuned variants.
• Inputs and Outputs:
  • Input: Text string, images (normalized to 896 x 896 resolution and encoded to 256 tokens each), with a total input context of 128K tokens for the 4B, 12B, and 27B sizes, and 32K tokens for the 1B size.
  • Output: Generated text in response to the input, with a total output context of 8192 tokens.

Model Data

• Training Dataset: These models were trained on a dataset of text data that includes web documents, code, mathematics, and images. The 27B model was trained with 14 trillion tokens, the 12B model with 12 trillion tokens, the 4B model with 4 trillion tokens, and the 1B model with 2 trillion tokens.
• Data Preprocessing: Key data cleaning and filtering methods include CSAM filtering, sensitive data filtering, and additional filtering based on content quality and safety.

Implementation Information

• Hardware: Gemma was trained using Tensor Processing Unit (TPU) hardware (TPUv4p, TPUv5p, and TPUv5e).
• Software: Training was done using JAX and ML Pathways.

📚 Documentation

Model Page

Gemma

Resources and Technical Documentation

• Gemma 3 Technical Report
• Responsible Generative AI Toolkit
• Gemma on Kaggle
• Gemma on Vertex Model Garden

Terms of Use

Terms

Authors

Google DeepMind

Citation

@article{gemma_2025,
    title={Gemma 3},
    url={https://goo.gle/Gemma3Report},
    publisher={Kaggle},
    author={Gemma Team},
    year={2025}
}

🔧 Technical Details

Benchmark Results

Reasoning and factuality

STEM and code

Multilingual

Multimodal

Ethics and Safety

• Evaluation Approach: Our evaluation methods include structured evaluations and internal red - teaming testing of relevant content policies. These models were evaluated against categories such as child safety, content safety, and representational harms.
• Evaluation Results: For all areas of safety testing, we saw major improvements relative to previous Gemma models. All testing was conducted without safety filters. A limitation was that only English language prompts were included.

Usage and Limitations

• Intended Usage: Open vision - language models (VLMs) have a wide range of applications, including content creation, chatbots, and text summarization.
• Limitations: Users should be aware of certain limitations of these models.

📄 License

The license is Gemma.