Instella 3B Stage1

🚀 Instella✨: Fully Open Language Models with Stellar Performance

AMD is thrilled to unveil Instella, a collection of cutting - edge, fully open 3 - billion - parameter language models (LMs) trained from scratch on AMD Instinct™ MI300X GPUs. These Instella models outshine existing fully open models of similar sizes and achieve competitive results when pitted against state - of - the - art open - weight models like Llama - 3.2 - 3B, Gemma - 2 - 2B, and Qwen - 2.5 - 3B, including their instruction - tuned counterparts.

Figure 1: Pareto frontier of pre - training tokens vs average performance for pre - trained and instruction - tuned models.

By training Instella from the ground up on Instinct MI300X GPUs, we showcase our hardware's capacity and scalability in handling demanding large - scale AI training workloads, presenting a viable option in the AI hardware market. In line with AMD's commitment to open source, we are releasing all artifacts related to Instella models [here](#additional - resources), such as model weights, detailed training configurations, datasets, and code. This enables the AI community to collaborate, replicate, and innovate, thereby accelerating progress.

✨ Features

• Announcing Instella: A series of 3 - billion - parameter language models developed by AMD, trained from scratch on 128 Instinct MI300X GPUs.
• Superior Performance: Instella models significantly outperform existing fully open LMs (Figure 1) of comparable size. They also bridge the gap between fully open and open - weight models by achieving competitive performance against state - of - the - art open - weight models and their instruction - tuned counterparts.
• Fully Open and Accessible: We are fully open - sourcing the model weights, training hyperparameters, datasets, and code, which will foster innovation and collaboration within the AI community.
• Efficient Training: Supported by the AMD ROCm software stack, Instella employs efficient training techniques like FlashAttention - 2, Torch Compile, and Fully Sharded Data Parallelism (FSDP) with hybrid sharding to scale model training over a large cluster.

📦 Instella Models

In this release, we present the following Instella models:

The Instella models are text - only, autoregressive transformer - based LMs with 3 billion parameters. Architecturally, Instella has 36 decoder layers, each with 32 attention heads. These models support a sequence length of up to 4,096 tokens and have a vocabulary size of ~50,000 tokens using the OLMo tokenizer. During both pre - training and fine - tuning, we used FlashAttention - 2, Torch Compile, and bfloat16 mixed - precision training to reduce memory usage, leading to computational speedups and optimal resource utilization. To balance inter - node memory efficiency and intra - node communication overhead within our cluster, we employed fully sharded data parallelism (FSDP) with hybrid sharding, where model parameters, gradients, and optimizer states are sharded within a node and replicated across the nodes.

Our training pipeline is based on the open - sourced OLMo codebase, adapted, and optimized for our hardware and model architecture. For pre - training, we used a total of 128 Instinct MI300X GPUs distributed across 16 nodes, with each node having 8x Instinct MI300X GPUs. We evaluated our models and baselines using standard tasks from OLMES, [FastChat MT - Bench](https://github.com/lm - sys/FastChat/blob/main/fastchat/llm_judge/README.md), and [Alpaca](https://github.com/tatsu - lab/alpaca_eval/tree/main). For more details about the architecture, training pipeline/hyperparameters, and evaluation results, please refer to our [Blog](https://rocm.blogs.amd.com/artificial - intelligence/introducing - instella - 3B/README.html), [Hugging Face model card](https://huggingface.co/amd/Instella - 3B), and [Github repository](https://github.com/AMD - AIG - AIMA/Instella).

🔧 Technical Details

Training Pipeline

The training of the Instella models consists of four stages, each incrementally enhancing the model's capabilities from basic natural language understanding to instruction following and alignment with human preferences.

Model Summary

Hyparparameter

Training Data

⚠️ Important Note

Further information concerning the training datasets, including applicable licensing terms and use restrictions, may be located at the linked source location.

🚀 Quick Start

📦 Installation

First, install PyTorch according to the instructions specific to your operating system. For AMD GPUs, you can also start with a rocm/pytorch docker.

To install from source (recommended for training/fine - tuning) run:

git clone https://github.com/AMD - AIG - AIMA/Instella.git
cd Instella
# install Flash - Attention on MI300X
GPU_ARCH=gfx942 MAX_JOBS=$(nproc) pip install git+https://github.com/Dao - AILab/flash - attention.git - v
# install other dependencies
pip install -e .[all]

💻 Usage Examples

Basic Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
checkpoint = "amd/Instella-3B-Instruct"

tokenizer = AutoTokenizer.from_pretrained(checkpoint, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(checkpoint, device_map="auto", trust_remote_code=True)

prompt = [{"role": "user", "content": "What are the benefits of open - source AI research?"}]
inputs = tokenizer.apply_chat_template(
    prompt,
    add_generation_prompt=True,
    return_tensors='pt'
)

tokens = model.generate(
    inputs.to(model.device),
    max_new_tokens=1024,
    temperature=0.8,
    do_sample=True
)

print(tokenizer.decode(tokens[0], skip_special_tokens=False))

Advanced Usage

You can also use the TRL CLI to chat with the model from the terminal:

pip install trl
trl chat --model_name_or_path amd/Instella-3B-Instruct --trust_remote_code --max_new_tokens 1024

# <root>:
# which is bigger 9.8 or 9.11?

# <amd/Instella-3B-Instruct>:
# 9.8 is bigger than 9.11. The difference between the two numbers is 0.69 (9.8 - 9.11 = 0.69), which indicates that 9.8 is 0.69 units larger than 9.11.  

📚 Documentation

Results

Pre - training

• Outstanding Performance: Both Instella - 3B - Stage1 and Instella - 3B models outperform all other fully open models in all benchmarks individually (except PIQA). Our final pre - trained checkpoint Instella - 3B outperforms the existing top - performing fully open pre - trained models by an average lead of ⬆️8.08%, with significant improvements in ARC Challenge [+8.02%], ARC Easy [+3.51%], Winnograde [+4.7%], OpenBookQA [+3.88%], MMLU [+13.12%] and ️GSM8K [+48.98%].
• Enhanced Performance through Second - stage Pre - training: Second - stage pre - training increased the overall average performance relative to stage - 1 by ⬆️5.26%, significantly narrowing the performance gap between the Instella - 3B model and closed - source models. It outperforms Llama - 3.2 - 3B by an average of ⬆️4.08% (+5.69% [ARC Challenge], +5.61% [ARC Easy], and +29.72% [GSM8k]), Gemma - 2 - 2B by an average of ⬆️7.25% (+13.38% [ARC Challenge], +11.23% [ARC Easy], +4.5% [Hellaswag], +7.6% [OpenBookQA], +5.03% [MMLU], and +32.45% [GSM8k]), and is competitive with Qwen - 2.5 - 3B in most benchmarks.
• Competitive Alternative: The multi - stage pre - training with diverse and high - quality data mix significantly enhanced Instella - 3B's capabilities, making it a competitive and open alternative in the landscape of comparable - size language models.

Instruction - tuning Results

• Dominant Performance: The Instella - 3B - Instruct model consistently outperforms other fully open models across all evaluated benchmarks, with an average score lead of ⬆️ 14.37% compared to the next top - performing fully open instruction - tuned models. It has substantial margins across all chat benchmarks (+13% [MMLU], 7.57% [TruthfulQA], 7.43% [BBH], +4.46% [GPQA], +37.15 [IFEval], 10.08% [Alpaca 2], and 1.2% [MT - Bench]).
• Narrowing the Gap: Instella - 3B - Instruct narrows the performance gap with leading open - weight models. It performs on par with or slightly surpasses existing state - of - the - art open - weight instruction - tuned models such as Llama - 3.2 - 3B - Instruct (+5.24% [TruthfulQA], 0.45% [GPQA], and +0.1% [MT - Bench]), and Qwen2.5 - 3B - Instruct (+2.01% [GPQA] and +8.87% [IFEval]), while significantly outperforming Gemma - 2 - 2B - Instruct with an average score lead of ⬆️5.83% (+0.55% [MMLU], +3.79 [BBH], +4.91 [GPQA], +20.47 [GSM8k], +19.98 [Minerva MATH], and +15.17% [IFEval]).
• Exceptional in Instruction - following and QA Tasks: Overall, Instella - 3B - Instruct excels in instruction - following tasks and multi - turn QA tasks like TruthfulQA, GPQA, IFEval, and MT - Bench. It is highly competitive compared to existing state - of - the - art open - weight models on other knowledge - recall and math benchmarks, despite being trained on significantly fewer training tokens.

📄 License

• The Instella - 3B models are licensed for academic and research purposes under a ResearchRAIL license.
• The [amd/Instella - GSM8K - synthetic](https://huggingface.co/datasets/amd/Instella - GSM8K - synthetic) dataset used in second - stage pre - training is built with Qwen2.5 - 72B - Instruct and is licensed for academic and research purposes under a ResearchRAIL license. Refer to the [LICENSE](https://huggingface.co/datasets/amd/Instella - GSM8K - synthetic/blob/main/LICENSE) and [NOTICES](https://huggingface.co/datasets/amd/Instella - GSM8K - synthetic/blob/main/NOTICES) in the [amd/Instella - GSM8K - synthetic](https://huggingface.co/datasets/amd/Instella - GSM8K - synthetic) dataset card files for more information.
• Refer to the [LICENSE](https://huggingface.co/amd/Instella - 3B/blob/main/LICENSE) and [NOTICES](https://huggingface.co/amd/Instella - 3B/blob/main/NOTICES) files for more information.

Conclusion

The release of the Instella family of models marks a significant step forward in advancing open - source AI and demonstrating the capabilities of AMD hardware in large - scale language model training. The 3 - billion - parameter models from the Instella family significantly outperform current fully open comparable - size models in key benchmarks and are competitive with comparable open - weight models. This is due to the high - quality data - mix selection, multi - stage training pipeline, and the use of high - performance Instinct MI300X GPUs for large - scale training.

By fully open - sourcing the Instella models, including weights, training configurations, datasets, and code, we aim to foster innovation and collaboration within the AI community. We believe that transparency, reproducibility, and accessibility are key drivers of progress in AI research and development. We invite developers, researchers, and AI enthusiasts to explore Instella, contribute to its continuous improvement, and join us in pushing the boundaries of what is possible with language models.

We will continue to enhance the models in multiple aspects, such as context length, reasoning ability, and multimodal capabilities. Additionally, we will scale up both the model and dataset while exploring diverse architectural approaches. Stay tuned for more exciting blogs on the Instella LMs family, its features, and capabilities!

Additional Resources

Hugging Face Model Cards

• Pre - trained models:
  • Instella - 3B - Stage1: [amd/Instella - 3B - Stage1](https://huggingface.co/amd/Instella - 3B - Stage1), First - stage pre - training checkpoint.
  • Instella - 3B: [amd/Instella - 3B](https://huggingface.co/amd/Instella - 3B), Final pre - training checkpoint.
• Instruction - tuned models:
  • Instella - 3B - SFT: [amd/Instella - 3B - SFT](https://huggingface.co/amd/Instella - 3B - SFT), Supervised fine - tuned checkpoint.
  • Instella - 3B - Instruct: [amd/Instella - 3B - Instruct](https://huggingface.co/amd/Instella - 3B - Instruct), Final Instruction - tuned checkpoint.

Datasets

Second - stage pre - training GSM8k synthetic dataset: [amd/Instella - GSM8K - synthetic](https://huggingface.co/datasets/amd/Instella - GSM8K - synthetic)

• The dataset consists of two splits: train and train_119K.
• For the Instella - 3B model's second - stage pre - training, we used the train_119K split, which is a subset of the larger train split.

Code

• Github: [https://github.com/AMD - AIG - AIMA/Instella](https://github.com/AMD - AIG - AIMA/Instella)

Please refer to the following blogs to get started with using these techniques on AMD GPUs:

• [PyTorch Fully Sharded Data Parallel (FSDP) on AMD GPUs with ROCm™](https://rocm.blogs.amd.com/artificial - intelligence/fsdp - training - pytorch/README.html)
• [Accelerating Large Language Models with Flash Attention on AMD GPUs](https://rocm.blogs.amd.com/artificial - intelligence/flash - attention/README.html)
• [Accelerate PyTorch Models using torch.compile on AMD GPUs with ROCm™](https://rocm.blogs.amd.com/artificial - intelligence/torch_compile/README.html)
• [Introducing the First AMD 1B Language Models: AMD OLMo](https://www.amd.com/en/developer/resources/technical - articles/introducing - the - first - amd - 1b - language - model.html)

Bias, Risks, and Limitations

⚠️ Important Note

• The models are being released for research purposes only and are not intended for use cases that require high levels of factuality, safety - critical situations, health, or medical applications, generating false information, or facilitating toxic conversations.
• Model checkpoints are made accessible without any safety promises. It is crucial for users to conduct comprehensive evaluations and implement safety filtering mechanisms as per their respective use cases.
• It may be possible to prompt the model to generate content that may be factually inaccurate, harmful, violent, toxic, biased, or otherwise objectionable. Such content may also get generated by prompts that did not intend to produce output as such. Users are thus requested to be aware of this and exercise caution and responsible thinking when using the model.
• The multi - lingual abilities of the models have not been tested and thus may misunderstand and generate erroneous responses across different languages.

Citations

Feel free to cite our Instella - 3B models:

@misc{Instella,
    title = {Instella: Fully Open Language Models with Stellar Performance},
    url = {https://huggingface.co/amd/Instella - 3B},
    author = {Jiang Liu, Jialian Wu, Xiaodong Yu, Prakamya Mishra, Sudhanshu Ranjan, Zicheng Liu, Chaitanya Manem, Yusheng Su, Pratik Prabhanjan Brahma, Gowtham Ramesh, Ximeng Sun, Ze Wang, Emad Barsoum},
    month = {March},
    year = {2025}
}