Qwen3 4B Llamafile

🚀 Qwen 3 4B - llamafile

Mozilla packaged the Qwen 3 models into executable weights called llamafiles, providing an easy and fast way to use the model on multiple operating systems.

🚀 Quick Start

Obtain the Model

To get started, you need both the Qwen 3 weights and the llamafile software, which are included in a single file. You can download and run it as follows:

wget https://huggingface.co/Mozilla/Qwen3-0.6B-llamafile/resolve/main/Qwen_Qwen3-4B-Q4_K_M.llamafile
chmod +x Qwen_Qwen3-4B-Q4_K_M.llamafile
./Qwen_Qwen3-4B-Q4_K_M.llamafile

The default mode of operation for these llamafiles is the new command-line chatbot interface.

✨ Features

Usage Modes

• Command-line Chatbot: You can use triple quotes to ask questions on multiple lines. Pass commands like /stats and /context to see runtime status information. Change the system prompt by passing the -p "new system prompt" flag. Press CTRL - C to interrupt the model and CTRL - D to exit.
• Web GUI: Use the --server mode to open a tab with a chatbot and completion interface in your browser. Pass the --help flag for additional help. The server also has an OpenAI API - compatible completions endpoint accessible via Python using the openai pip package.

./Qwen_Qwen3-4B-Q4_K_M.llamafile --server

• Advanced CLI Mode: Useful for shell scripting. Pass the --cli flag to use it. Pass the --help flag for additional help.

./Qwen_Qwen3-4B-Q4_K_M.llamafile --cli -p 'four score and seven' --log-disable

Context Window

This model has a max context window size of 128k tokens. By default, a context window size of 8192 tokens is used. Pass the -c 0 flag to use the maximum context size, which is big enough for a small book. Use the -f book.txt flag if you want to have a conversation with your book.

GPU Acceleration

• NVIDIA or AMD GPUs: On GPUs with sufficient RAM, pass the -ngl 999 flag to use the system's NVIDIA or AMD GPU(s).
  • Windows (NVIDIA): Only the graphics card driver needs to be installed.
  • Windows (AMD): Install the ROCm SDK v6.1 and pass the flags --recompile --gpu amd the first time you run your llamafile.
• NVIDIA GPUs (Performance Optimization): By default, the pre - built tinyBLAS library is used for matrix multiplications. If you have the CUDA SDK installed, pass the --recompile flag to build a GGML CUDA library using cuBLAS for maximum performance.

📚 Documentation

Troubleshooting

• General Troubleshooting: See the "Gotchas" section of the README.
• Linux: To avoid run - detector errors, install the APE interpreter:

sudo wget -O /usr/bin/ape https://cosmo.zip/pub/cosmos/bin/ape-$(uname -m).elf
sudo chmod +x /usr/bin/ape
sudo sh -c "echo ':APE:M::MZqFpD::/usr/bin/ape:' >/proc/sys/fs/binfmt_misc/register"
sudo sh -c "echo ':APE-jart:M::jartsr::/usr/bin/ape:' >/proc/sys/fs/binfmt_misc/register"

• Windows: There's a 4GB limit on executable sizes.

About llamafile

llamafile is a new format introduced by Mozilla on Nov 20th, 2023. It uses Cosmopolitan Libc to turn LLM weights into runnable llama.cpp binaries that run on the stock installs of six OSes for both ARM64 and AMD64.

🚀 Qwen3 - 4B

✨ Features of Qwen3

Qwen3 is the latest generation of large language models in the Qwen series, offering a comprehensive suite of dense and mixture - of - experts (MoE) models. It has the following key features:

• Seamless Mode Switching: Uniquely support seamless switching between thinking mode (for complex logical reasoning, math, and coding) and non - thinking mode (for efficient, general - purpose dialogue) within a single model, ensuring optimal performance across various scenarios.
• Enhanced Reasoning: Significantly enhanced reasoning capabilities, surpassing previous QwQ (in thinking mode) and Qwen2.5 instruct models (in non - thinking mode) on mathematics, code generation, and commonsense logical reasoning.
• Human Preference Alignment: Superior human preference alignment, excelling in creative writing, role - playing, multi - turn dialogues, and instruction following, to deliver a more natural, engaging, and immersive conversational experience.
• Agent Capabilities: Expertise in agent capabilities, enabling precise integration with external tools in both thinking and unthinking modes and achieving leading performance among open - source models in complex agent - based tasks.
• Multilingual Support: Support of 100+ languages and dialects with strong capabilities for multilingual instruction following and translation.

📦 Model Overview

For more details, including benchmark evaluation, hardware requirements, and inference performance, please refer to the blog, GitHub, and Documentation.

⚠️ Important Note

If you encounter significant endless repetitions, please refer to the Best Practices section for optimal sampling parameters, and set the presence_penalty to 1.5.

💻 Usage Examples

Basic Usage

The code of Qwen3 has been included in the latest Hugging Face transformers. It is recommended to use the latest version of transformers.

from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "Qwen/Qwen3-4B"

# load the tokenizer and the model
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype="auto",
    device_map="auto"
)

# prepare the model input
prompt = "Give me a short introduction to large language model."
messages = [
    {"role": "user", "content": prompt}
]
text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
    enable_thinking=True # Switches between thinking and non-thinking modes. Default is True.
)
model_inputs = tokenizer([text], return_tensors="pt").to(model.device)

# conduct text completion
generated_ids = model.generate(
    **model_inputs,
    max_new_tokens=32768
)
output_ids = generated_ids[0][len(model_inputs.input_ids[0]):].tolist() 

# parsing thinking content
try:
    # rindex finding 151668 (</think>)
    index = len(output_ids) - output_ids[::-1].index(151668)
except ValueError:
    index = 0

thinking_content = tokenizer.decode(output_ids[:index], skip_special_tokens=True).strip("\n")
content = tokenizer.decode(output_ids[index:], skip_special_tokens=True).strip("\n")

print("thinking content:", thinking_content)
print("content:", content)

Advanced Usage - Deployment

You can use sglang>=0.4.6.post1 or vllm>=0.8.5 to create an OpenAI - compatible API endpoint:

• SGLang:

python -m sglang.launch_server --model-path Qwen/Qwen3-4B --reasoning-parser qwen3

• vLLM:

vllm serve Qwen/Qwen3-4B --enable-reasoning --reasoning-parser deepseek_r1

For local use, applications such as Ollama, LMStudio, MLX - LM, llama.cpp, and KTransformers have also supported Qwen3.

Switching Between Thinking and Non - Thinking Mode

💡 Usage Tip

The enable_thinking switch is also available in APIs created by SGLang and vLLM. Please refer to the documentation for SGLang and vLLM users.

enable_thinking=True

By default, Qwen3 has thinking capabilities enabled, similar to QwQ - 32B. This means the model will use its reasoning abilities to enhance the quality of generated responses.

text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
    enable_thinking=True  # True is the default value for enable_thinking
)

In this mode, the model will generate think content wrapped in a <think>...</think> block, followed by the final response.

⚠️ Important Note

For thinking mode, use Temperature = 0.6, TopP = 0.95, TopK = 20, and MinP = 0 (the default setting in generation_config.json). DO NOT use greedy decoding, as it can lead to performance degradation and endless repetitions. For more detailed guidance, please refer to the Best Practices section.

enable_thinking=False

We provide a hard switch to strictly disable the model's thinking behavior, aligning its functionality with the previous Qwen2.5 - Instruct models. This mode is particularly useful in scenarios where disabling thinking is essential for enhancing efficiency.

text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
    enable_thinking=False  # Setting enable_thinking=False disables thinking mode
)

In this mode, the model will not generate any think content and will not include a <think>...</think> block.

⚠️ Important Note

For non - thinking mode, we suggest using Temperature = 0.7, TopP = 0.8, TopK = 20, and MinP = 0. For more detailed guidance, please refer to the Best Practices section.

Advanced Usage: Switching Between Thinking and Non - Thinking Modes via User Input

We provide a soft switch mechanism that allows users to dynamically control the model's behavior when enable_thinking=True. Specifically, you can add /think and /no_think to user prompts or system messages to switch the model's thinking mode from turn to turn. The model will follow the most recent instruction in multi - turn conversations.

from transformers import AutoModelForCausalLM, AutoTokenizer

class QwenChatbot:
    def __init__(self, model_name="Qwen/Qwen3-4B"):
        self.tokenizer = AutoTokenizer.from_pretrained(model_name)
        self.model = AutoModelForCausalLM.from_pretrained(model_name)
        self.history = []

    def generate_response(self, user_input):
        messages = self.history + [{"role": "user", "content": user_input}]

        text = self.tokenizer.apply_chat_template(
            messages,
            tokenize=False,
            add_generation_prompt=True
        )

        inputs = self.tokenizer(text, return_tensors="pt")
        response_ids = self.model.generate(**inputs, max_new_tokens=32768)[0][len(inputs.input_ids[0]):].tolist()
        response = self.tokenizer.decode(response_ids, skip_special_tokens=True)

        # Update history
        self.history.append({"role": "user", "content": user_input})
        self.history.append({"role": "assistant", "content": response})

        return response

# Example Usage
if __name__ == "__main__":
    chatbot = QwenChatbot()

    # First input (without /think or /no_think tags, thinking mode is enabled by default)
    user_input_1 = "How many r's in strawberries?"
    print(f"User: {user_input_1}")
    response_1 = chatbot.generate_response(user_input_1)
    print(f"Bot: {response_1}")
    print("----------------------")

    # Second input with /no_think
    user_input_2 = "Then, how many r's in blueberries? /no_think"
    print(f"User: {user_input_2}")
    response_2 = chatbot.generate_response(user_input_2)
    print(f"Bot: {response_2}") 
    print("----------------------")

    # Third input with /think
    # (The rest of the code continues as in the original)

📄 License

This project is licensed under the Apache - 2.0 license.