Qwen3 8B Base

🚀 Qwen3-8B-Base

Qwen3-8B-Base is a powerful causal language model from the Qwen series, offering high - performance language processing capabilities with a large context length.

🚀 Quick Start

The code of Qwen3 has been integrated into the latest Hugging Face transformers. It is recommended to use the latest version of transformers.

If you use transformers<4.51.0, you will encounter the following error:

KeyError: 'qwen3'

✨ Features

Qwen3 Highlights

Qwen3 is the latest generation of large language models in the Qwen series, offering a comprehensive suite of dense and mixture - of - experts (MoE) models. Building upon extensive advancements in training data, model architecture, and optimization techniques, Qwen3 delivers the following key improvements over the previously released Qwen2.5:

• Expanded Higher - Quality Pre - training Corpus: Qwen3 is pre - trained on 36 trillion tokens across 119 languages — tripling the language coverage of Qwen2.5 — with a much richer mix of high - quality data, including coding, STEM, reasoning, book, multilingual, and synthetic data.
• Training Techniques and Model Architecture: Qwen3 incorporates a series of training techniques and architectural refinements, including global - batch load balancing loss for MoE models and qk layernorm for all models, leading to improved stability and overall performance.
• Three - stage Pre - training: Stage 1 focuses on broad language modeling and general knowledge acquisition, Stage 2 improves reasoning skills like STEM, coding, and logical reasoning, and Stage 3 enhances long - context comprehension by extending training sequence lengths up to 32k tokens.
• Scaling Law Guided Hyperparameter Tuning: Through comprehensive scaling law studies across the three - stage pre - training pipeline, Qwen3 systematically tunes critical hyperparameters — such as learning rate scheduler and batch size — separately for dense and MoE models, resulting in better training dynamics and final performance across different model scales.

Model Overview

Qwen3 - 8B - Base has the following features:

For more details, including benchmark evaluation, hardware requirements, and inference performance, please refer to our blog, GitHub, and Documentation.

📚 Documentation

Detailed evaluation results are reported in this 📑 blog.

Citation

If you find our work helpful, feel free to give us a cite.

@misc{qwen3,
    title  = {Qwen3},
    url    = {https://qwenlm.github.io/blog/qwen3/},
    author = {Qwen Team},
    month  = {April},
    year   = {2025}
}

📄 License

This project is licensed under the Apache - 2.0 license.