Qwen3-4B-GGUF Open-Source Large Language Model - Free Deployment, Excellent Inference and Multilingual Capabilities

Qwen3 4B GGUF

Developed by prithivMLmods

Qwen3 is the latest generation of large language models in the Tongyi Qianwen series, offering a complete combination of dense models and Mixture of Experts (MoE) models. Based on large-scale training, Qwen3 achieves breakthrough progress in reasoning capabilities, instruction following, agent functions, and multilingual support.

Large Language Model EnglishOpen Source License:Apache-2.0 #Mixture of Experts Model #Multilingual Text Generation #Efficient Quantized Inference

Downloads 829

Release Time : 5/13/2025

Model Overview

Qwen3-4B is a large language model with 4B parameters based on the Qwen3 architecture, supporting text generation tasks and providing various quantized versions to meet different hardware requirements.

Model Features

Mixture of Experts Model

Offers a complete combination of dense models and Mixture of Experts (MoE) models, optimizing inference efficiency.

Multi-Precision Support

Provides multiple precision versions from FP32 to 3-bit quantization to accommodate different hardware and performance needs.

High-Performance Inference

Achieves breakthrough progress in reasoning capabilities, instruction following, and agent functions.

Model Capabilities

Text Generation

Instruction Following

Agent Functions

Multilingual Support

Use Cases

Natural Language Processing

Text Generation

Generates coherent and logical text content.

Intelligent Dialogue

Builds intelligent dialogue systems supporting multi-turn interactions.

🚀 Qwen3-4B-GGUF

Qwen3 is the latest generation of large language models in the Qwen series. It offers a comprehensive suite of dense and mixture - of - experts (MoE) models. Through extensive training, it brings groundbreaking advancements in reasoning, instruction - following, agent capabilities, and multilingual support.

📚 Documentation

Model Files

Property	Details
Model Type	Qwen3-4B-GGUF
Base Model	Qwen/Qwen3-4B
Pipeline Tag	text-generation
Library Name	transformers
License	apache-2.0

Model Files Table

File Name	Size	Quantization	Format	Description
`Qwen3_4B.F32.gguf`	16.1 GB	FP32	GGUF	Full precision (float32) version
`Qwen3_4B.BF16.gguf`	8.05 GB	BF16	GGUF	BFloat16 precision version
`Qwen3_4B.F16.gguf`	8.05 GB	FP16	GGUF	Float16 precision version
`Qwen3_4B.Q3_K_M.gguf`	2.08 GB	Q3_K_M	GGUF	3 - bit quantized (K M variant)
`Qwen3_4B.Q3_K_S.gguf`	1.89 GB	Q3_K_S	GGUF	3 - bit quantized (K S variant)
`Qwen3_4B.Q4_K_M.gguf`	2.5 GB	Q4_K_M	GGUF	4 - bit quantized (K M variant)
`Qwen3_4B.Q4_K_S.gguf`	2.38 GB	Q4_K_S	GGUF	4 - bit quantized (K S variant)
`Qwen3_4B.Q5_K_M.gguf`	2.89 GB	Q5_K_M	GGUF	5 - bit quantized (K M variant)
`Qwen3_4B.Q8_0.gguf`	4.28 GB	Q8_0	GGUF	8 - bit quantized
`.gitattributes`	2.02 kB	—	—	Git LFS tracking file
`config.json`	31 B	—	—	Configuration placeholder
`README.md`	3.6 kB	—	—	Model documentation

Quants Usage

(sorted by size, not necessarily quality. IQ - quants are often preferable over similar sized non - IQ quants)

Link	Type	Size/GB	Notes
GGUF	Q2_K	0.4
GGUF	Q3_K_S	0.5
GGUF	Q3_K_M	0.5	lower quality
GGUF	Q3_K_L	0.5
GGUF	IQ4_XS	0.6
GGUF	Q4_K_S	0.6	fast, recommended
GGUF	Q4_K_M	0.6	fast, recommended
GGUF	Q5_K_S	0.6
GGUF	Q5_K_M	0.7
GGUF	Q6_K	0.7	very good quality
GGUF	Q8_0	0.9	fast, best quality
GGUF	f16	1.6	16 bpw, overkill

Here is a handy graph by ikawrakow comparing some lower - quality quant types (lower is better):

📄 License

This project is licensed under the Apache - 2.0 license.

Featured Recommended AI Models

Empowering the Future, Your AI Solution Knowledge Base

English 简体中文繁體中文にほんご