Gemma 3 12b It Qat Q4 0 Gguf

🚀 Gemma 3 model card

Gemma 3 is a multimodal model from Google, capable of handling text and image input and generating text output. It has a large context window, multilingual support, and is suitable for various text generation and image understanding tasks.

Model Page: Gemma

⚠️ Important Note

This repository corresponds to the 12B instruction-tuned version of the Gemma 3 model in GGUF format using Quantization Aware Training (QAT). The GGUF corresponds to Q4_0 quantization. Thanks to QAT, the model is able to preserve similar quality as bfloat16 while significantly reducing the memory requirements to load the model. You can find the half-precision version here.

Resources and Technical Documentation:

• Gemma 3 Technical Report
• Responsible Generative AI Toolkit
• Gemma on Kaggle
• Gemma on Vertex Model Garden

Terms of Use: Terms

Authors: Google DeepMind

✨ Features

Gemma is a family of lightweight, state-of-the-art open models from Google, built from the same research and technology used to create the Gemini models. Gemma 3 models are multimodal, handling text and image input and generating text output, with open weights for both pre-trained variants and instruction-tuned variants. Key features include:

• A large, 128K context window.
• Multilingual support in over 140 languages.
• Availability in more sizes than previous versions.
• Suitability for a variety of text generation and image understanding tasks, such as question answering, summarization, and reasoning.
• Ability to be deployed in environments with limited resources like laptops, desktops, or personal cloud infrastructure.

📦 Installation

To access Gemma on Hugging Face, you’re required to review and agree to Google’s usage license. To do this, please ensure you’re logged in to Hugging Face and click below. Requests are processed immediately.

💻 Usage Examples

Basic Usage

llama.cpp (text-only)

./llama-cli -hf google/gemma-3-12b-it-qat-q4_0-gguf -p "Write a poem about the Kraken."

llama.cpp (image input)

wget https://github.com/bebechien/gemma/blob/main/surprise.png?raw=true -O ~/Downloads/surprise.png
./llama-gemma3-cli -hf google/gemma-3-12b-it-qat-q4_0-gguf -p "Describe this image." --image ~/Downloads/surprise.png

ollama (text only)

Using GGUFs with Ollama via Hugging Face does not support image inputs at the moment. Please check the docs on running gated repositories.

ollama run hf.co/google/gemma-3-12b-it-qat-q4_0-gguf

📚 Documentation

Model Information

Description

Gemma is a family of lightweight, state-of-the-art open models from Google, built from the same research and technology used to create the Gemini models. Gemma 3 models are multimodal, handling text and image input and generating text output, with open weights for both pre-trained variants and instruction-tuned variants. They are well-suited for a variety of text generation and image understanding tasks, and their relatively small size allows for deployment in resource-limited environments.

Inputs and outputs

• Input:
  • Text string, such as a question, a prompt, or a document to be summarized.
  • Images, normalized to 896 x 896 resolution and encoded to 256 tokens each.
  • Total input context of 128K tokens for the 4B, 12B, and 27B sizes, and 32K tokens for the 1B size.
• Output:
  • Generated text in response to the input, such as an answer to a question, analysis of image content, or a summary of a document.
  • Total output context of 8192 tokens.

Model Data

Training Dataset

These models were trained on a diverse dataset of text data from various sources. The 27B model was trained with 14 trillion tokens, the 12B model with 12 trillion tokens, the 4B model with 4 trillion tokens, and the 1B model with 2 trillion tokens. Key components include:

• Web Documents: A diverse collection of web text in over 140 languages, exposing the model to a broad range of linguistic styles, topics, and vocabulary.
• Code: Helps the model learn programming language syntax and patterns, improving its code generation and understanding abilities.
• Mathematics: Enables the model to learn logical reasoning, symbolic representation, and address mathematical queries.
• Images: Allows the model to perform image analysis and visual data extraction tasks.

Data Preprocessing

Key data cleaning and filtering methods applied to the training data include:

• CSAM Filtering: Rigorous filtering at multiple stages to exclude harmful and illegal content.
• Sensitive Data Filtering: Automated techniques to filter out personal information and other sensitive data.
• Additional methods: Filtering based on content quality and safety in line with our policies.

Implementation Information

Hardware

Gemma was trained using Tensor Processing Unit (TPU) hardware (TPUv4p, TPUv5p, and TPUv5e). TPUs offer several advantages for training vision-language models:

• Performance: Specifically designed to handle massive computations, speeding up training compared to CPUs.
• Memory: Often come with large amounts of high-bandwidth memory, allowing for handling large models and batch sizes during training, leading to better model quality.
• Scalability: TPU Pods provide a scalable solution for handling large foundation models, enabling distributed training across multiple devices.
• Cost-effectiveness: Can be a more cost-effective solution for training large models, especially considering time and resource savings.
• Alignment with Google's commitments to operate sustainably.

Software

Training was done using JAX and ML Pathways. JAX allows for faster and more efficient training of large models on the latest hardware, including TPUs. ML Pathways is Google's effort to build artificially intelligent systems capable of generalizing across multiple tasks, suitable for foundation models.

Evaluation

Benchmark Results

These models were evaluated against a large collection of different datasets and metrics to cover different aspects of text generation:

Reasoning and factuality

STEM and code

Multilingual

Multimodal

Ethics and Safety

Evaluation Approach

Our evaluation methods include structured evaluations and internal red-teaming testing of relevant content policies. Red-teaming was conducted by multiple teams with different goals and human evaluation metrics. Categories evaluated for ethics and safety include:

• Child Safety: Evaluation of text-to-text and image-to-text prompts covering child safety policies, including child sexual abuse and exploitation.
• Content Safety: Evaluation of text-to-text and image-to-text prompts covering safety policies such as harassment, violence, gore, and hate speech.
• Representational Harms: Evaluation of text-to-text and image-to-text prompts covering safety policies including bias, stereotyping, and harmful associations or inaccuracies.

In addition to development level evaluations, we conduct "assurance evaluations" for responsibility governance decision making. These are separate from the model development team and inform release decisions. High-level findings are fed back to the model team, and prompt sets are held out to prevent overfitting.

Evaluation Results

For all areas of safety testing, we saw major improvements in child safety, content safety, and representational harms compared to previous Gemma models. All testing was conducted without safety filters to evaluate model capabilities and behaviors. The model produced minimal policy violations and showed significant improvements over previous models' performance with respect to ungrounded inferences. A limitation of our evaluations was the use of only English language prompts.

Usage and Limitations

Intended Usage

Open vision-language models (VLMs) have a wide range of applications across various industries and domains. Potential uses include:

• Content Creation and Communication
  • Text Generation: Generate creative text formats such as poems, scripts, code, marketing copy, and email drafts.
  • Chatbots and Conversational AI: Power conversational interfaces for customer service, virtual assistants, or interactive applications.
  • Text Summarization: Generate concise summaries of text.
  • Image Data Extraction: Extract, interpret, and summarize visual data for text communications.
• Research and Education
  • Natural Language Processing (NLP) and VLM Research: Serve as a foundation for research and algorithm development.
  • Language Learning Tools: Support interactive language learning experiences.
  • Knowledge Exploration: Assist in exploring large bodies of text.

Limitations

• Training Data: The quality and diversity of training data can influence model capabilities, and biases or gaps in the data can lead to limitations in responses.
• Context and Task Complexity: Models perform better on tasks with clear prompts, and performance can be affected by the amount of context provided.
• Language Ambiguity and Nuance: Natural language complexity can make it difficult for models to grasp subtle nuances.
• Factual Accuracy: Models may generate incorrect or outdated factual statements as they are not knowledge bases.
• Common Sense: Models may lack the ability to apply common sense reasoning in certain situations.

Citation

@article{gemma_2025,
    title={Gemma 3},
    url={https://goo.gle/Gemma3Report},
    publisher={Kaggle},
    author={Gemma Team},
    year={2025}
}

📄 License

The license for this model is gemma.