Qwen3 0.6B Llamafile

🚀 Qwen 3 0.6B - llamafile

Mozilla packaged the Qwen 3 models into executable weights (llamafiles), providing an easy and fast way to use the model across multiple operating systems.

🚀 Quick Start

To get started, you need both the Qwen 3 weights and the llamafile software. Both are included in a single file, which can be downloaded and run as follows:

wget https://huggingface.co/Mozilla/Qwen3-0.6B-llamafile/resolve/main/Qwen_Qwen3-0.6B-Q4_K_M.llamafile
chmod +x Qwen_Qwen3-0.6B-Q4_K_M.llamafile
./Qwen_Qwen3-0.6B-Q4_K_M.llamafile

The default mode of operation for these llamafiles is the new command line chatbot interface.

✨ Features

• Model Creator: Qwen
• Original Model: Qwen/Qwen3-0.6B
• Packaging: Mozilla packaged the Qwen 3 models into llamafiles, enabling easy use on Linux, MacOS, Windows, FreeBSD, OpenBSD, and NetBSD systems (both AMD64 and ARM64).
• Software Update: Software Last Updated: 2025-05-14
• Llamafile Version: Llamafile Version: 0.9.3

💻 Usage Examples

Basic Usage

You can use triple quotes to ask questions on multiple lines. You can pass commands like /stats and /context to see runtime status information. You can change the system prompt by passing the -p "new system prompt" flag. You can press CTRL - C to interrupt the model, and CTRL - D to exit.

If you prefer to use a web GUI, then a --server mode is provided, which will open a tab with a chatbot and completion interface in your browser. For additional help, pass the --help flag. The server also has an OpenAI API compatible completions endpoint that can be accessed via Python using the openai pip package.

./Qwen_Qwen3-0.6B-Q4_K_M.llamafile --server

Advanced Usage

An advanced CLI mode is provided that's useful for shell scripting. You can use it by passing the --cli flag. For additional help, pass the --help flag.

./Qwen_Qwen3-0.6B-Q4_K_M.llamafile --cli -p 'four score and seven' --log-disable

📚 Documentation

Troubleshooting

• Linux: To avoid run - detector errors on Linux, install the APE interpreter:

sudo wget -O /usr/bin/ape https://cosmo.zip/pub/cosmos/bin/ape-$(uname -m).elf
sudo chmod +x /usr/bin/ape
sudo sh -c "echo ':APE:M::MZqFpD::/usr/bin/ape:' >/proc/sys/fs/binfmt_misc/register"
sudo sh -c "echo ':APE-jart:M::jartsr::/usr/bin/ape:' >/proc/sys/fs/binfmt_misc/register"

• Windows: There's a 4GB limit on executable sizes.

Context Window

This model has a max context window size of 128k tokens. By default, a context window size of 8192 tokens is used. You can ask llamafile to use the maximum context size by passing the -c 0 flag. If you want to have a conversation with a book, you can use the -f book.txt flag.

GPU Acceleration

• Sufficient RAM GPUs: On GPUs with sufficient RAM, the -ngl 999 flag may be passed to use the system's NVIDIA or AMD GPU(s).
• Windows - NVIDIA: On Windows, only the graphics card driver needs to be installed if you own an NVIDIA GPU.
• Windows - AMD: On Windows, if you have an AMD GPU, you should install the ROCm SDK v6.1 and then pass the flags --recompile --gpu amd the first time you run your llamafile.
• NVIDIA GPUs - Performance: On NVIDIA GPUs, by default, the prebuilt tinyBLAS library is used for matrix multiplications. If you have the CUDA SDK installed, you can pass the --recompile flag to build a GGML CUDA library using cuBLAS for maximum performance.

For further information, please see the llamafile README.

About llamafile

llamafile is a new format introduced by Mozilla on Nov 20th 2023. It uses Cosmopolitan Libc to turn LLM weights into runnable llama.cpp binaries that run on the stock installs of six OSes for both ARM64 and AMD64.

🚀 Qwen3 - 0.6B

✨ Features

Qwen3 Highlights

Qwen3 is the latest generation of large language models in the Qwen series, offering dense and mixture - of - experts (MoE) models. It delivers advancements in reasoning, instruction - following, agent capabilities, and multilingual support:

• Dual Modes: Uniquely supports seamless switching between thinking mode (for complex logical reasoning, math, and coding) and non - thinking mode (for efficient, general - purpose dialogue) within a single model.
• Enhanced Reasoning: Significantly enhances reasoning capabilities, surpassing previous QwQ (in thinking mode) and Qwen2.5 instruct models (in non - thinking mode) in mathematics, code generation, and commonsense logical reasoning.
• Human Preference Alignment: Excels in creative writing, role - playing, multi - turn dialogues, and instruction following, providing a more natural, engaging, and immersive conversational experience.
• Agent Capabilities: Expertise in agent capabilities, enabling precise integration with external tools in both thinking and non - thinking modes and achieving leading performance among open - source models in complex agent - based tasks.
• Multilingual Support: Supports 100+ languages and dialects with strong capabilities for multilingual instruction following and translation.

Model Overview

For more details, including benchmark evaluation, hardware requirements, and inference performance, please refer to our blog, GitHub, and Documentation.

⚠️ Important Note

If you encounter significant endless repetitions, please refer to the Best Practices section for optimal sampling parameters, and set the presence_penalty to 1.5.

🚀 Quick Start

The code of Qwen3 has been included in the latest Hugging Face transformers. We advise you to use the latest version of transformers.

With transformers<4.51.0, you will encounter the following error:

KeyError: 'qwen3'

The following is a code snippet showing how to use the model to generate content based on given inputs:

from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "Qwen/Qwen3-0.6B"

# load the tokenizer and the model
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype="auto",
    device_map="auto"
)

# prepare the model input
prompt = "Give me a short introduction to large language model."
messages = [
    {"role": "user", "content": prompt}
]
text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
    enable_thinking=True # Switches between thinking and non-thinking modes. Default is True.
)
model_inputs = tokenizer([text], return_tensors="pt").to(model.device)

# conduct text completion
generated_ids = model.generate(
    **model_inputs,
    max_new_tokens=32768
)
output_ids = generated_ids[0][len(model_inputs.input_ids[0]):].tolist() 

# parsing thinking content
try:
    # rindex finding 151668 (</think>)
    index = len(output_ids) - output_ids[::-1].index(151668)
except ValueError:
    index = 0

thinking_content = tokenizer.decode(output_ids[:index], skip_special_tokens=True).strip("\n")
content = tokenizer.decode(output_ids[index:], skip_special_tokens=True).strip("\n")

print("thinking content:", thinking_content)
print("content:", content)

For deployment, you can use sglang>=0.4.6.post1 or vllm>=0.8.5 to create an OpenAI - compatible API endpoint:

• SGLang:

  python -m sglang.launch_server --model-path Qwen/Qwen3-0.6B --reasoning-parser qwen3
  

• vLLM:

  vllm serve Qwen/Qwen3-0.6B --enable-reasoning --reasoning-parser deepseek_r1
  

For local use, applications such as Ollama, LMStudio, MLX - LM, llama.cpp, and KTransformers have also supported Qwen3.

💻 Usage Examples

Switching Between Thinking and Non - Thinking Mode

💡 Usage Tip

The enable_thinking switch is also available in APIs created by SGLang and vLLM. Please refer to our documentation for SGLang and vLLM users.

enable_thinking=True

By default, Qwen3 has thinking capabilities enabled, similar to QwQ - 32B. This means the model will use its reasoning abilities to enhance the quality of generated responses.

text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
    enable_thinking=True  # True is the default value for enable_thinking
)

In this mode, the model will generate think content wrapped in a <think>...</think> block, followed by the final response.

⚠️ Important Note

For thinking mode, use Temperature = 0.6, TopP = 0.95, TopK = 20, and MinP = 0 (the default setting in generation_config.json). DO NOT use greedy decoding, as it can lead to performance degradation and endless repetitions. For more detailed guidance, please refer to the Best Practices section.

enable_thinking=False

We provide a hard switch to strictly disable the model's thinking behavior, aligning its functionality with the previous Qwen2.5 - Instruct models. This mode is particularly useful in scenarios where disabling thinking is essential for enhancing efficiency.

text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
    enable_thinking=False  # Setting enable_thinking=False disables thinking mode
)

In this mode, the model will not generate any think content and will not include a <think>...</think> block.

⚠️ Important Note

For non - thinking mode, we suggest using Temperature = 0.7, TopP = 0.8, TopK = 20, and MinP = 0. For more detailed guidance, please refer to the Best Practices section.

Advanced Usage: Switching Between Thinking and Non - Thinking Modes via User Input

We provide a soft switch mechanism that allows users to dynamically control the model's behavior when enable_thinking=True. Specifically, you can add /think and /no_think to user prompts or system messages to switch the model's thinking mode from turn to turn. The model will follow the most recent instruction in multi - turn conversations.

Here is an example of a multi - turn conversation:

from transformers import AutoModelForCausalLM, AutoTokenizer

class QwenChatbot:
    def __init__(self, model_name="Qwen/Qwen3-0.6B"):
        self.tokenizer = AutoTokenizer.from_pretrained(model_name)
        self.model = AutoModelForCausalLM.from_pretrained(model_name)
        self.history = []

    def generate_response(self, user_input):
        messages = self.history + [{"role": "user", "content": user_input}]

        text = self.tokenizer.apply_chat_template(
            messages,
            tokenize=False,
            add_generation_prompt=True
        )

        inputs = self.tokenizer(text, return_tensors="pt")
        response_ids = self.model.generate(**inputs, max_new_tokens=32768)[0][len(inputs.input_ids[0]):].tolist()
        response = self.tokenizer.decode(response_ids, skip_special_tokens=True)

        # Update history
        self.history.append({"role": "user", "content": user_input})
        self.history.append({"role": "assistant", "content": response})

        return response

# Example Usage
if __name__ == "__main__":
    chatbot = QwenChatbot()

    # First input (without /think or /no_think tags, thinking mode is enabled by default)
    user_input_1 = "How many r's in strawberries?"
    print(f"User: {user_input_1}")
    response_1 = chatbot.generate_response(user_input_1)
    print(f"Bot: {response_1}")
    print("----------------------")

    # Second input with /no_think
    user_input_2 = "Then, how many r's in blueberries? /no_think"
    print(f"User: {user_input_2}")
    response_2 = chatbot.generate_response(user_input_2)
    print(f"Bot: {response_2}") 
    print("----------------------")

    # Third input with /think
    user_input_3 = "Really? /think"
    print(f"User: {user_input_3}")
    response_3 = chatbot.generate_response(user_input_3)
    print(f"Bot: {response_3}")

📄 License

This project is licensed under the Apache - 2.0 license.