Gemma-3-4b-it-GGUF Open Source Model - Free Deployment for Image-Text to Text Conversion

Gemma 3 4b It GGUF

Developed by second-state

Gemma-3-4b-it-GGUF is a quantized version of Google's Gemma-3-4b-it model, enabling it to run on LlamaEdge and suitable for image-text to text conversion tasks.

Transformers

#Multimodal text generation #High-precision quantization #Large context support

Downloads 2,120

Release Time : 3/12/2025

Model Overview

This project provides quantization processing for the Gemma-3-4b-it model, enabling it to run efficiently on the LlamaEdge platform and supporting image-text to text conversion tasks.

Model Features

Efficient quantization

Provides multiple quantization versions to meet different hardware and performance requirements, ranging from Q2_K to Q8_0.

LlamaEdge compatibility

Optimized specifically for the LlamaEdge platform and can run efficiently on edge devices.

Image-text conversion

Supports converting image content into text, suitable for multimodal tasks.

Model Capabilities

Image-text conversion

Multimodal processing

Edge computing

Use Cases

Multimodal applications

Image description generation

Convert the input image content into descriptive text.

Generate high-quality image description text.

Visual question answering

Answer user questions based on the image content.

Accurately answer questions related to the image content.

Edge computing

Mobile device image processing

Process images in real-time on mobile devices and generate text descriptions.

Low-latency image-text conversion.

🚀 Gemma-3-4b-it-GGUF

This project provides a quantized version of the gemma-3-4b-it model in GGUF format, enabling efficient inference with LlamaEdge.

🚀 Quick Start

Original Model

The original model can be found at google/gemma-3-4b-it.

Run with LlamaEdge

LlamaEdge version: v0.18.5 and above.

Prompt template:

Prompt type: gemma-3

Prompt string:

<bos><start_of_turn>user
{user_message}<end_of_turn>
<start_of_turn>model
{model_message}<end_of_turn>model

Context size: 128000

Run as LlamaEdge service:

Chat:

wasmedge --dir .:. --nn-preload default:GGML:AUTO:gemma-3-4b-it-Q5_K_M.gguf \
  llama-api-server.wasm \
  --prompt-template gemma-3 \
  --ctx-size 128000 \
  --model-name gemma-3-4b

Images:

⚠️ Important Note

Input images are required to be normalized to 896 x 896 resolution and encoded to 256 tokens each.

wasmedge --dir .:. --nn-preload default:GGML:AUTO:gemma-3-4b-it-Q5_K_M.gguf \
  llama-api-server.wasm \
  --prompt-template gemma-3 \
  --llava-mmproj gemma-3-4b-it-mmproj-f16.gguf \
  --ctx-size 128000 \
  --model-name gemma-3-4b

Run as LlamaEdge command app:

wasmedge --dir .:. \
  --nn-preload default:GGML:AUTO:gemma-3-4b-it-Q5_K_M.gguf \
  llama-chat.wasm \
  --prompt-template gemma-3 \
  --ctx-size 128000

📦 Installation

The installation mainly involves running the commands provided in the "Run with LlamaEdge" section. Make sure you have the necessary dependencies installed, such as wasmedge and the appropriate LlamaEdge version.

📚 Documentation

Quantized GGUF Models

Property	Details
Model Type	Quantized GGUF models of `gemma-3-4b-it`
Training Data	Not provided

The following table lists the available quantized GGUF models:

Name	Quant method	Bits	Size	Use case
gemma-3-4b-it-Q2_K.gguf	Q2_K	2	1.73 GB	smallest, significant quality loss - not recommended for most purposes
gemma-3-4b-it-Q3_K_L.gguf	Q3_K_L	3	2.24 GB	small, substantial quality loss
gemma-3-4b-it-Q3_K_M.gguf	Q3_K_M	3	2.10 GB	very small, high quality loss
gemma-3-4b-it-Q3_K_S.gguf	Q3_K_S	3	1.94 GB	very small, high quality loss
gemma-3-4b-it-Q4_0.gguf	Q4_0	4	2.36 GB	legacy; small, very high quality loss - prefer using Q3_K_M
gemma-3-4b-it-Q4_K_M.gguf	Q4_K_M	4	2.49 GB	medium, balanced quality - recommended
gemma-3-4b-it-Q4_K_S.gguf	Q4_K_S	4	2.38 GB	small, greater quality loss
gemma-3-4b-it-Q5_0.gguf	Q5_0	5	2.76 GB	legacy; medium, balanced quality - prefer using Q4_K_M
gemma-3-4b-it-Q5_K_M.gguf	Q5_K_M	5	2.83 GB	large, very low quality loss - recommended
gemma-3-4b-it-Q5_K_S.gguf	Q5_K_S	5	2.76 GB	large, low quality loss - recommended
gemma-3-4b-it-Q6_K.gguf	Q6_K	6	3.19 GB	very large, extremely low quality loss
gemma-3-4b-it-Q8_0.gguf	Q8_0	8	4.13 GB	very large, extremely low quality loss - not recommended
gemma-3-4b-it-f16.gguf	f16	16	7.77 GB
gemma-3-4b-it-mmproj-f16.gguf	f16	16	851 MB

Quantized with llama.cpp b4875

📄 License

The model is under the gemma license.

Featured Recommended AI Models

Empowering the Future, Your AI Solution Knowledge Base

English 简体中文繁體中文にほんご