Model Overview
Model Features
Model Capabilities
Use Cases
đ Qwen3-0.6B-GPTQ-Int8
Qwen3-0.6B-GPTQ-Int8 is a powerful language model with unique features such as seamless mode - switching, enhanced reasoning, and excellent agent capabilities, offering high - quality text generation.
đ Quick Start
The code of Qwen3 has been integrated into the latest Hugging Face transformers
. We highly recommend using the latest version of transformers
.
If you use transformers<4.51.0
, you'll encounter the following error:
KeyError: 'qwen3'
The following code snippet shows how to use the model to generate content based on given inputs:
from transformers import AutoModelForCausalLM, AutoTokenizer
model_name = "Qwen/Qwen3-0.6B-GPTQ-Int8"
# load the tokenizer and the model
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
model_name,
torch_dtype="auto",
device_map="auto"
)
# prepare the model input
prompt = "Give me a short introduction to large language model."
messages = [
{"role": "user", "content": prompt}
]
text = tokenizer.apply_chat_template(
messages,
tokenize=False,
add_generation_prompt=True,
enable_thinking=True # Switches between thinking and non - thinking modes. Default is True.
)
model_inputs = tokenizer([text], return_tensors="pt").to(model.device)
# conduct text completion
generated_ids = model.generate(
**model_inputs,
max_new_tokens=32768
)
output_ids = generated_ids[0][len(model_inputs.input_ids[0]):].tolist()
# parsing thinking content
try:
# rindex finding 151668 (</think>)
index = len(output_ids) - output_ids[::-1].index(151668)
except ValueError:
index = 0
thinking_content = tokenizer.decode(output_ids[:index], skip_special_tokens=True).strip("\n")
content = tokenizer.decode(output_ids[index:], skip_special_tokens=True).strip("\n")
print("thinking content:", thinking_content)
print("content:", content)
For deployment, you can use sglang>=0.4.6.post1
or vllm==0.8.4
to create an OpenAI - compatible API endpoint:
- SGLang:
python -m sglang.launch_server --model - path Qwen/Qwen3-0.6B-GPTQ-Int8 --reasoning - parser qwen3
- vLLM:
vllm serve Qwen/Qwen3-0.6B-GPTQ-Int8 --enable - reasoning --reasoning - parser deepseek_r1
For more usage guides, please refer to our GPTQ documentation.
â ī¸ Important Note
If you encounter significant endless repetitions, please refer to the [Best Practices](#best - practices) section for optimal sampling parameters, and set the
presence_penalty
to 1.5.
⨠Features
Qwen3 is the latest generation of large language models in the Qwen series, offering a comprehensive suite of dense and mixture - of - experts (MoE) models. After extensive training, Qwen3 has made groundbreaking advancements in reasoning, instruction - following, agent capabilities, and multilingual support, with the following key features:
- Unique seamless switching: It supports seamless switching between thinking mode (for complex logical reasoning, math, and coding) and non - thinking mode (for efficient, general - purpose dialogue) within a single model, ensuring optimal performance across various scenarios.
- Enhanced reasoning capabilities: It has significantly enhanced reasoning capabilities, surpassing previous QwQ (in thinking mode) and Qwen2.5 instruct models (in non - thinking mode) in mathematics, code generation, and commonsense logical reasoning.
- Superior human preference alignment: It excels in creative writing, role - playing, multi - turn dialogues, and instruction following, delivering a more natural, engaging, and immersive conversational experience.
- Expertise in agent capabilities: It has excellent agent capabilities, enabling precise integration with external tools in both thinking and non - thinking modes and achieving leading performance among open - source models in complex agent - based tasks.
- Multilingual support: It supports 100+ languages and dialects with strong capabilities for multilingual instruction following and translation.
đ Documentation
Model Overview
Qwen3-0.6B has the following features:
Property | Details |
---|---|
Model Type | Causal Language Models |
Training Stage | Pretraining & Post - training |
Number of Parameters | 0.6B |
Number of Paramaters (Non - Embedding) | 0.44B |
Number of Layers | 28 |
Number of Attention Heads (GQA) | 16 for Q and 8 for KV |
Context Length | 32,768 |
Quantization | GPTQ 8 - bit |
For more details, including benchmark evaluation, hardware requirements, and inference performance, please refer to our blog, GitHub, and Documentation.
Switching Between Thinking and Non - Thinking Mode
đĄ Usage Tip
The
enable_thinking
switch is also available in APIs created by SGLang and vLLM. Please refer to our documentation for [SGLang](https://qwen.readthedocs.io/en/latest/deployment/sglang.html#thinking - non - thinking - modes) and [vLLM](https://qwen.readthedocs.io/en/latest/deployment/vllm.html#thinking - non - thinking - modes) users.
Basic Usage
enable_thinking=True
By default, Qwen3 has thinking capabilities enabled, similar to QwQ - 32B. This means the model will use its reasoning abilities to enhance the quality of generated responses. For example, when explicitly setting enable_thinking=True
or leaving it as the default value in tokenizer.apply_chat_template
, the model will engage its thinking mode.
text = tokenizer.apply_chat_template(
messages,
tokenize=False,
add_generation_prompt=True,
enable_thinking=True # True is the default value for enable_thinking
)
In this mode, the model will generate think content wrapped in a <think>...</think>
block, followed by the final response.
â ī¸ Important Note
For thinking mode, use
Temperature = 0.6
,TopP = 0.95
,TopK = 20
, andMinP = 0
(the default setting ingeneration_config.json
). DO NOT use greedy decoding, as it can lead to performance degradation and endless repetitions. For more detailed guidance, please refer to the [Best Practices](#best - practices) section.
enable_thinking=False
We provide a hard switch to strictly disable the model's thinking behavior, aligning its functionality with the previous Qwen2.5 - Instruct models. This mode is particularly useful in scenarios where disabling thinking is essential for enhancing efficiency.
text = tokenizer.apply_chat_template(
messages,
tokenize=False,
add_generation_prompt=True,
enable_thinking=False # Setting enable_thinking=False disables thinking mode
)
In this mode, the model will not generate any think content and will not include a <think>...</think>
block.
â ī¸ Important Note
For non - thinking mode, we suggest using
Temperature = 0.7
,TopP = 0.8
,TopK = 20
, andMinP = 0
. For more detailed guidance, please refer to the [Best Practices](#best - practices) section.
Advanced Usage: Switching Between Thinking and Non - Thinking Modes via User Input
We provide a soft switch mechanism that allows users to dynamically control the model's behavior when enable_thinking=True
. Specifically, you can add /think
and /no_think
to user prompts or system messages to switch the model's thinking mode from turn to turn. The model will follow the most recent instruction in multi - turn conversations.
Here is an example of a multi - turn conversation:
from transformers import AutoModelForCausalLM, AutoTokenizer
class QwenChatbot:
def __init__(self, model_name="Qwen/Qwen3-0.6B-GPTQ-Int8"):
self.tokenizer = AutoTokenizer.from_pretrained(model_name)
self.model = AutoModelForCausalLM.from_pretrained(model_name)
self.history = []
def generate_response(self, user_input):
messages = self.history + [{"role": "user", "content": user_input}]
text = self.tokenizer.apply_chat_template(
messages,
tokenize=False,
add_generation_prompt=True
)
inputs = self.tokenizer(text, return_tensors="pt")
response_ids = self.model.generate(**inputs, max_new_tokens=32768)[0][len(inputs.input_ids[0]):].tolist()
response = self.tokenizer.decode(response_ids, skip_special_tokens=True)
# Update history
self.history.append({"role": "user", "content": user_input})
self.history.append({"role": "assistant", "content": response})
return response
# Example Usage
if __name__ == "__main__":
chatbot = QwenChatbot()
# First input (without /think or /no_think tags, thinking mode is enabled by default)
user_input_1 = "How many r's in strawberries?"
print(f"User: {user_input_1}")
response_1 = chatbot.generate_response(user_input_1)
print(f"Bot: {response_1}")
print("----------------------")
# Second input with /no_think
user_input_2 = "Then, how many r's in blueberries? /no_think"
print(f"User: {user_input_2}")
response_2 = chatbot.generate_response(user_input_2)
print(f"Bot: {response_2}")
print("----------------------")
# Third input with /think
user_input_3 = "Really? /think"
print(f"User: {user_input_3}")
response_3 = chatbot.generate_response(user_input_3)
print(f"Bot: {response_3}")
â ī¸ Important Note
For API compatibility, when
enable_thinking=True
, regardless of whether the user uses/think
or/no_think
, the model will always output a block wrapped in<think>...</think>
. However, the content inside this block may be empty if thinking is disabled. Whenenable_thinking=False
, the soft switches are not valid. Regardless of any/think
or/no_think
tags input by the user, the model will not generate think content and will not include a<think>...</think>
block.
Agentic Use
Qwen3 excels in tool - calling capabilities. We recommend using [Qwen - Agent](https://github.com/QwenLM/Qwen - Agent) to make the best use of the agentic ability of Qwen3. Qwen - Agent encapsulates tool - calling templates and tool - calling parsers internally, greatly reducing coding complexity.
To define the available tools, you can use the MCP configuration file, use the integrated tool of Qwen - Agent, or integrate other tools by yourself.
from qwen_agent.agents import Assistant
# Define LLM
llm_cfg = {
'model': 'Qwen/Qwen3-0.6B-GPTQ-Int8',
# Use the endpoint provided by Alibaba Model Studio:
# 'model_type': 'qwen_dashscope',
# 'api_key': os.getenv('DASHSCOPE_API_KEY'),
# Use a custom endpoint compatible with OpenAI API:
'model_server': 'http://localhost:8000/v1', # api_base
'api_key': 'EMPTY',
# Other parameters:
# 'generate_cfg': {
# # Add: When the response content is `<think>this is the thought</think>this is the answer;
# # Do not add: When the response has been separated by reasoning_content and content.
# 'thought_in_content': True,
# },
}
# Define Tools
tools = [
{'mcpServers': { # You can specify the MCP configuration file
'time': {
'command': 'uvx',
'args': ['mcp-server-time', '--local-timezone=Asia/Shanghai']
},
"fetch": {
"command": "uvx",
"args": ["mcp-server-fetch"]
}
}
},
'code_interpreter', # Built - in tools
]
# Define Agent
bot = Assistant(llm=llm_cfg, function_list=tools)
# Streaming generation
messages = [{'role': 'user', 'content': 'https://qwenlm.github.io/blog/ Introduce the latest developments of Qwen'}]
for responses in bot.run(messages=messages):
pass
print(responses)
Performance
Mode | QUANTIZATION TYPE | LiveBench 2024 - 11 - 25 | GPQA | MMLU - Redux |
---|---|---|---|---|
Thinking | bf16 | 30.3 | 27.9 | 55.6 |
Thinking | GPTQ - int8 | 30.2 | 29.3 | 53.8 |
Non - Thinking | bf16 | 21.8 | 22.9 | 44.6 |
Non - Thinking | GPTQ - int8 | 21.8 | 23.2 | 44.9 |
Best Practices
To achieve optimal performance, we recommend the following settings:
- Sampling Parameters:
- For thinking mode (
enable_thinking=True
), useTemperature = 0.6
,TopP = 0.95
,TopK = 20
, andMinP = 0
. DO NOT use greedy decoding, as it can lead to performance degradation and endless repetitions. - For non - thinking mode (
enable_thinking=False
), we suggest usingTemperature = 0.7
,TopP = 0.8
,TopK = 20
, andMinP = 0
. - For supported frameworks, you can adjust the
presence_penalty
parameter between 0 and 2 to reduce endless repetitions. We strongly recommend setting this value to 1.5 for quantized models. However, using a higher value may occasionally result in language mixing and a slight decrease in model performance.
- For thinking mode (
- Adequate Output Length: We recommend using an output length of 32,768 tokens for most queries.
đ License
This project is licensed under the Apache - 2.0 license.

