Typhoon2.1 Gemma3 12b Gguf

🚀 Typhoon2.1-Gemma3-12B

Typhoon2.1-Gemma3-12B is a Thai large language model (instruct) based on Gemma3 12B, offering 12 billion parameters, a 128K context length, and function-calling capabilities.

✨ Features

• Thai Instruct Model: Typhoon2.1-Gemma3-12B is an instruct Thai large language model with 12 billion parameters.
• Long Context Length: It supports a 128K context length.
• Function-Calling Capabilities: The model comes with function-calling capabilities.
• Quantization: This repo contains the gguf q4_k_m quantization of the original Typhoon2.1 12B.
• Text-Only Model: It is a text-only model.

🚀 Quick Start

Performance

Use with llama.cpp

Installation

Install llama.cpp through brew (works on Mac and Linux):

brew install llama.cpp

Usage

You can invoke the llama.cpp server or the CLI.

CLI

llama-cli --hf-repo scb10x/typhoon2.1-gemma3-12b-gguf --hf-file typhoon2.1-gemma3-12b-q4_k_m.gguf -p "The meaning to life and the universe is"

Server

llama-server --hf-repo scb10x/typhoon2.1-gemma3-12b-gguf --hf-file typhoon2.1-gemma3-12b-q4_k_m.gguf -c 2048

Note: You can also use this checkpoint directly through the usage steps listed in the Llama.cpp repo.

Manual Steps

1. Clone llama.cpp from GitHub:

git clone https://github.com/ggerganov/llama.cpp

2. Move into the llama.cpp folder and build it with LLAMA_CURL = 1 flag along with other hardware-specific flags (e.g., LLAMA_CUDA = 1 for Nvidia GPUs on Linux):

cd llama.cpp && LLAMA_CURL = 1 make

3. Run inference through the main binary:

./llama-cli --hf-repo scb10x/typhoon2.1-gemma3-12b-gguf --hf-file typhoon2.1-gemma3-12b-q4_k_m.gguf -p "The meaning to life and the universe is"

or

./llama-server --hf-repo scb10x/typhoon2.1-gemma3-12b-gguf --hf-file typhoon2.1-gemma3-12b-q4_k_m.gguf -c 2048