Qwen3 0.6B GPTQ Int8

🚀 Qwen3-0.6B-GPTQ-Int8

Qwen3-0.6B-GPTQ-Int8 is a powerful language model with unique features such as seamless mode - switching, enhanced reasoning, and excellent agent capabilities, offering high - quality text generation.

🚀 Quick Start

The code of Qwen3 has been integrated into the latest Hugging Face transformers. We highly recommend using the latest version of transformers.

If you use transformers<4.51.0, you'll encounter the following error:

KeyError: 'qwen3'

The following code snippet shows how to use the model to generate content based on given inputs:

from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "Qwen/Qwen3-0.6B-GPTQ-Int8"

# load the tokenizer and the model
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype="auto",
    device_map="auto"
)

# prepare the model input
prompt = "Give me a short introduction to large language model."
messages = [
    {"role": "user", "content": prompt}
]
text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
    enable_thinking=True # Switches between thinking and non - thinking modes. Default is True.
)
model_inputs = tokenizer([text], return_tensors="pt").to(model.device)

# conduct text completion
generated_ids = model.generate(
    **model_inputs,
    max_new_tokens=32768
)
output_ids = generated_ids[0][len(model_inputs.input_ids[0]):].tolist() 

# parsing thinking content
try:
    # rindex finding 151668 (</think>)
    index = len(output_ids) - output_ids[::-1].index(151668)
except ValueError:
    index = 0

thinking_content = tokenizer.decode(output_ids[:index], skip_special_tokens=True).strip("\n")
content = tokenizer.decode(output_ids[index:], skip_special_tokens=True).strip("\n")

print("thinking content:", thinking_content)
print("content:", content)

For deployment, you can use sglang>=0.4.6.post1 or vllm==0.8.4 to create an OpenAI - compatible API endpoint:

• SGLang:

  python -m sglang.launch_server --model - path Qwen/Qwen3-0.6B-GPTQ-Int8 --reasoning - parser qwen3
  

• vLLM:

  vllm serve Qwen/Qwen3-0.6B-GPTQ-Int8 --enable - reasoning --reasoning - parser deepseek_r1
  

For more usage guides, please refer to our GPTQ documentation.

⚠️ Important Note

If you encounter significant endless repetitions, please refer to the [Best Practices](#best - practices) section for optimal sampling parameters, and set the presence_penalty to 1.5.

✨ Features

Qwen3 is the latest generation of large language models in the Qwen series, offering a comprehensive suite of dense and mixture - of - experts (MoE) models. After extensive training, Qwen3 has made groundbreaking advancements in reasoning, instruction - following, agent capabilities, and multilingual support, with the following key features:

• Unique seamless switching: It supports seamless switching between thinking mode (for complex logical reasoning, math, and coding) and non - thinking mode (for efficient, general - purpose dialogue) within a single model, ensuring optimal performance across various scenarios.
• Enhanced reasoning capabilities: It has significantly enhanced reasoning capabilities, surpassing previous QwQ (in thinking mode) and Qwen2.5 instruct models (in non - thinking mode) in mathematics, code generation, and commonsense logical reasoning.
• Superior human preference alignment: It excels in creative writing, role - playing, multi - turn dialogues, and instruction following, delivering a more natural, engaging, and immersive conversational experience.
• Expertise in agent capabilities: It has excellent agent capabilities, enabling precise integration with external tools in both thinking and non - thinking modes and achieving leading performance among open - source models in complex agent - based tasks.
• Multilingual support: It supports 100+ languages and dialects with strong capabilities for multilingual instruction following and translation.

📚 Documentation

Model Overview

Qwen3-0.6B has the following features:

For more details, including benchmark evaluation, hardware requirements, and inference performance, please refer to our blog, GitHub, and Documentation.

Switching Between Thinking and Non - Thinking Mode

💡 Usage Tip

The enable_thinking switch is also available in APIs created by SGLang and vLLM. Please refer to our documentation for [SGLang](https://qwen.readthedocs.io/en/latest/deployment/sglang.html#thinking - non - thinking - modes) and [vLLM](https://qwen.readthedocs.io/en/latest/deployment/vllm.html#thinking - non - thinking - modes) users.

Basic Usage

enable_thinking=True

By default, Qwen3 has thinking capabilities enabled, similar to QwQ - 32B. This means the model will use its reasoning abilities to enhance the quality of generated responses. For example, when explicitly setting enable_thinking=True or leaving it as the default value in tokenizer.apply_chat_template, the model will engage its thinking mode.

text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
    enable_thinking=True  # True is the default value for enable_thinking
)

In this mode, the model will generate think content wrapped in a <think>...</think> block, followed by the final response.

⚠️ Important Note

For thinking mode, use Temperature = 0.6, TopP = 0.95, TopK = 20, and MinP = 0 (the default setting in generation_config.json). DO NOT use greedy decoding, as it can lead to performance degradation and endless repetitions. For more detailed guidance, please refer to the [Best Practices](#best - practices) section.

enable_thinking=False

We provide a hard switch to strictly disable the model's thinking behavior, aligning its functionality with the previous Qwen2.5 - Instruct models. This mode is particularly useful in scenarios where disabling thinking is essential for enhancing efficiency.

text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
    enable_thinking=False  # Setting enable_thinking=False disables thinking mode
)

In this mode, the model will not generate any think content and will not include a <think>...</think> block.

⚠️ Important Note

For non - thinking mode, we suggest using Temperature = 0.7, TopP = 0.8, TopK = 20, and MinP = 0. For more detailed guidance, please refer to the [Best Practices](#best - practices) section.

Advanced Usage: Switching Between Thinking and Non - Thinking Modes via User Input

We provide a soft switch mechanism that allows users to dynamically control the model's behavior when enable_thinking=True. Specifically, you can add /think and /no_think to user prompts or system messages to switch the model's thinking mode from turn to turn. The model will follow the most recent instruction in multi - turn conversations.

Here is an example of a multi - turn conversation:

from transformers import AutoModelForCausalLM, AutoTokenizer

class QwenChatbot:
    def __init__(self, model_name="Qwen/Qwen3-0.6B-GPTQ-Int8"):
        self.tokenizer = AutoTokenizer.from_pretrained(model_name)
        self.model = AutoModelForCausalLM.from_pretrained(model_name)
        self.history = []

    def generate_response(self, user_input):
        messages = self.history + [{"role": "user", "content": user_input}]

        text = self.tokenizer.apply_chat_template(
            messages,
            tokenize=False,
            add_generation_prompt=True
        )

        inputs = self.tokenizer(text, return_tensors="pt")
        response_ids = self.model.generate(**inputs, max_new_tokens=32768)[0][len(inputs.input_ids[0]):].tolist()
        response = self.tokenizer.decode(response_ids, skip_special_tokens=True)

        # Update history
        self.history.append({"role": "user", "content": user_input})
        self.history.append({"role": "assistant", "content": response})

        return response

# Example Usage
if __name__ == "__main__":
    chatbot = QwenChatbot()

    # First input (without /think or /no_think tags, thinking mode is enabled by default)
    user_input_1 = "How many r's in strawberries?"
    print(f"User: {user_input_1}")
    response_1 = chatbot.generate_response(user_input_1)
    print(f"Bot: {response_1}")
    print("----------------------")

    # Second input with /no_think
    user_input_2 = "Then, how many r's in blueberries? /no_think"
    print(f"User: {user_input_2}")
    response_2 = chatbot.generate_response(user_input_2)
    print(f"Bot: {response_2}") 
    print("----------------------")

    # Third input with /think
    user_input_3 = "Really? /think"
    print(f"User: {user_input_3}")
    response_3 = chatbot.generate_response(user_input_3)
    print(f"Bot: {response_3}")

⚠️ Important Note

For API compatibility, when enable_thinking=True, regardless of whether the user uses /think or /no_think, the model will always output a block wrapped in <think>...</think>. However, the content inside this block may be empty if thinking is disabled. When enable_thinking=False, the soft switches are not valid. Regardless of any /think or /no_think tags input by the user, the model will not generate think content and will not include a <think>...</think> block.

Agentic Use

Qwen3 excels in tool - calling capabilities. We recommend using [Qwen - Agent](https://github.com/QwenLM/Qwen - Agent) to make the best use of the agentic ability of Qwen3. Qwen - Agent encapsulates tool - calling templates and tool - calling parsers internally, greatly reducing coding complexity.

To define the available tools, you can use the MCP configuration file, use the integrated tool of Qwen - Agent, or integrate other tools by yourself.

from qwen_agent.agents import Assistant

# Define LLM
llm_cfg = {
    'model': 'Qwen/Qwen3-0.6B-GPTQ-Int8',

    # Use the endpoint provided by Alibaba Model Studio:
    # 'model_type': 'qwen_dashscope',
    # 'api_key': os.getenv('DASHSCOPE_API_KEY'),

    # Use a custom endpoint compatible with OpenAI API:
    'model_server': 'http://localhost:8000/v1',  # api_base
    'api_key': 'EMPTY',

    # Other parameters:
    # 'generate_cfg': {
    #         # Add: When the response content is `<think>this is the thought</think>this is the answer;
    #         # Do not add: When the response has been separated by reasoning_content and content.
    #         'thought_in_content': True,
    #     },
}

# Define Tools
tools = [
    {'mcpServers': {  # You can specify the MCP configuration file
            'time': {
                'command': 'uvx',
                'args': ['mcp-server-time', '--local-timezone=Asia/Shanghai']
            },
            "fetch": {
                "command": "uvx",
                "args": ["mcp-server-fetch"]
            }
        }
    },
  'code_interpreter',  # Built - in tools
]

# Define Agent
bot = Assistant(llm=llm_cfg, function_list=tools)

# Streaming generation
messages = [{'role': 'user', 'content': 'https://qwenlm.github.io/blog/ Introduce the latest developments of Qwen'}]
for responses in bot.run(messages=messages):
    pass
print(responses)

Performance

Best Practices

To achieve optimal performance, we recommend the following settings:

1. Sampling Parameters:
   • For thinking mode (enable_thinking=True), use Temperature = 0.6, TopP = 0.95, TopK = 20, and MinP = 0. DO NOT use greedy decoding, as it can lead to performance degradation and endless repetitions.
   • For non - thinking mode (enable_thinking=False), we suggest using Temperature = 0.7, TopP = 0.8, TopK = 20, and MinP = 0.
   • For supported frameworks, you can adjust the presence_penalty parameter between 0 and 2 to reduce endless repetitions. We strongly recommend setting this value to 1.5 for quantized models. However, using a higher value may occasionally result in language mixing and a slight decrease in model performance.
2. Adequate Output Length: We recommend using an output length of 32,768 tokens for most queries.

📄 License

This project is licensed under the Apache - 2.0 license.