Gemma 3 4b It Abliterated Q4 0 GGUF

🚀 BernTheCreator/gemma-3-4b-it-abliterated-Q4_0-GGUF

Combining the Abliterated Q4_0 - GGUF with a better mmproj (vision) option (x - ray_alpha), for a smoother experience.

This model was converted to GGUF format from mlabonne/gemma-3-4b-it-abliterated using llama.cpp via the ggml.ai's GGUF-my-repo space.

Refer to the original model card for more details on the model.

🚀 Quick Start

Prerequisites

• Base models:
  • mlabonne/gemma-3-4b-it-abliterated
  • SicariusSicariiStuff/X-Ray_Alpha
• Library: transformers
• License: gemma
• Pipeline tag: image-text-to-text
• Tags: llama-cpp, gguf-my-repo

📦 Installation

Install llama.cpp through brew (works on Mac and Linux):

brew install llama.cpp

💻 Usage Examples

Using llama.cpp CLI

llama-cli --hf-repo BernTheCreator/gemma-3-4b-it-abliterated-Q4_0-GGUF --hf-file gemma-3-4b-it-abliterated-q4_0.gguf -p "The meaning to life and the universe is"

Using llama.cpp Server

llama-server --hf-repo BernTheCreator/gemma-3-4b-it-abliterated-Q4_0-GGUF --hf-file gemma-3-4b-it-abliterated-q4_0.gguf -c 2048

Alternative Usage Steps

Step 1: Clone llama.cpp from GitHub

git clone https://github.com/ggerganov/llama.cpp

Step 2: Build llama.cpp

Move into the llama.cpp folder and build it with LLAMA_CURL = 1 flag along with other hardware - specific flags (for ex: LLAMA_CUDA = 1 for Nvidia GPUs on Linux).

cd llama.cpp && LLAMA_CURL=1 make

Step 3: Run inference

./llama-cli --hf-repo BernTheCreator/gemma-3-4b-it-abliterated-Q4_0-GGUF --hf-file gemma-3-4b-it-abliterated-q4_0.gguf -p "The meaning to life and the universe is"

or

./llama-server --hf-repo BernTheCreator/gemma-3-4b-it-abliterated-Q4_0-GGUF --hf-file gemma-3-4b-it-abliterated-q4_0.gguf -c 2048

📄 License

This project uses the gemma license.