Instella 3B

🚀 Instella✨: Fully Open Language Models with Stellar Performance

Instella is a family of fully open state-of-the-art 3-billion-parameter language models (LMs) developed by AMD. These models are trained from scratch on AMD Instinct™ MI300X GPUs, outperforming existing fully open models of similar sizes and achieving competitive performance compared to state-of-the-art open-weight models.

🚀 Quick Start

Installation

First, install PyTorch according to the instructions specific to your operating system. For AMD GPUs, you can also start with a rocm/pytorch docker.

To install from source (recommended for training/fine-tuning) run:

git clone https://github.com/AMD-AIG-AIMA/Instella.git
cd Instella
# install Flash-Attention on MI300X
GPU_ARCH=gfx942 MAX_JOBS=$(nproc) pip install git+https://github.com/Dao-AILab/flash-attention.git -v
# install other dependencies
pip install -e .[all]

Example Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
checkpoint = "amd/Instella-3B-Instruct"

tokenizer = AutoTokenizer.from_pretrained(checkpoint, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(checkpoint, device_map="auto", trust_remote_code=True)

prompt = [{"role": "user", "content": "What are the benefits of open-source AI research?"}]
inputs = tokenizer.apply_chat_template(
    prompt,
    add_generation_prompt=True,
    return_tensors='pt'
)

tokens = model.generate(
    inputs.to(model.device),
    max_new_tokens=1024,
    temperature=0.8,
    do_sample=True
)

print(tokenizer.decode(tokens[0], skip_special_tokens=False))

Chat in TRL

You can also use the TRL CLI to chat with the model from the terminal:

pip install trl
trl chat --model_name_or_path amd/Instella-3B-Instruct --trust_remote_code --max_new_tokens 1024

# <root>:
# which is bigger 9.8 or 9.11?

# <amd/Instella-3B-Instruct>:
# 9.8 is bigger than 9.11. The difference between the two numbers is 0.69 (9.8 - 9.11 = 0.69), which indicates that 9.8 is 0.69 units larger than 9.11.  

✨ Features

• High-performance models: Instella models significantly outperform existing fully open LMs of comparable size and bridge the gap between fully open and open weight models.
• Fully open and accessible: The model weights, training hyperparameters, datasets, and code are fully open-sourced, fostering innovation and collaboration within the AI community.
• Efficient training techniques: Instella employs efficient training techniques such as FlashAttention - 2, Torch Compile, and Fully Sharded Data Parallelism (FSDP) with hybrid sharding to scale model training over a large cluster.

📦 Installation

First, install PyTorch according to the instructions specific to your operating system. For AMD GPUs, you can also start with a rocm/pytorch docker.

To install from source (recommended for training/fine-tuning) run:

git clone https://github.com/AMD-AIG-AIMA/Instella.git
cd Instella
# install Flash-Attention on MI300X
GPU_ARCH=gfx942 MAX_JOBS=$(nproc) pip install git+https://github.com/Dao-AILab/flash-attention.git -v
# install other dependencies
pip install -e .[all]

💻 Usage Examples

Basic Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
checkpoint = "amd/Instella-3B-Instruct"

tokenizer = AutoTokenizer.from_pretrained(checkpoint, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(checkpoint, device_map="auto", trust_remote_code=True)

prompt = [{"role": "user", "content": "What are the benefits of open-source AI research?"}]
inputs = tokenizer.apply_chat_template(
    prompt,
    add_generation_prompt=True,
    return_tensors='pt'
)

tokens = model.generate(
    inputs.to(model.device),
    max_new_tokens=1024,
    temperature=0.8,
    do_sample=True
)

print(tokenizer.decode(tokens[0], skip_special_tokens=False))

Advanced Usage

pip install trl
trl chat --model_name_or_path amd/Instella-3B-Instruct --trust_remote_code --max_new_tokens 1024

# <root>:
# which is bigger 9.8 or 9.11?

# <amd/Instella-3B-Instruct>:
# 9.8 is bigger than 9.11. The difference between the two numbers is 0.69 (9.8 - 9.11 = 0.69), which indicates that 9.8 is 0.69 units larger than 9.11.  

📚 Documentation

Instella Models

In this release, the following Instella models are introduced:

The Instella models are text-only, autoregressive transformer-based LMs having 3 billion parameters. They support a sequence length of up to 4,096 tokens and have a vocabulary size of ~50,000 tokens using the OLMo tokenizer.

Training Pipeline

The training of the Instella models comprised of four stages, where each stage incrementally enhanced the model’s capabilities from fundamental natural language understanding to instruction following and alignment towards human preferences.

Model Summary

Hyparparameter

Results

Pre-training

• Both Instella - 3B - Stage1 & Instella - 3B models outperform all the other fully open models over all the benchmarks individually (except PIQA). The final pre - trained checkpoint Instella - 3B outperforms the existing top - performant fully open pre - trained models by a lead of ⬆️8.08% on average, with significant improvements in multiple benchmarks.
• Second stage pre - training elevated the overall average performance relative to stage - 1 by ⬆️5.26%, substantially narrowing the performance gap between Instella - 3B model vs the closed - source models, and outperforming Llama - 3.2 - 3B and Gemma - 2 - 2B on average.

Instruction - tuning Results

• The Instella - 3B - Instruct model consistently outperforms other fully open models across all evaluated benchmarks with a significant average score lead.
• It narrows the performance gap with leading open - weight models and performs on par with or slightly surpasses existing state - of - the - art open weight instruction - tuned models.

Training Data

⚠️ Important Note

Further information concerning the training datasets, including applicable licensing terms and use restrictions, may be located at the linked source location.

🔧 Technical Details

The Instella models are text - only, autoregressive transformer - based LMs having 3 billion parameters. Architecture - wise, Instella is packed with 36 decoder layers, each having 32 attention heads. These models support a sequence length of up to 4,096 tokens and have a vocabulary size of ~50,000 tokens using the OLMo tokenizer.

During both pre - training and fine - tuning, FlashAttention - 2, Torch Compile, and bfloat16 mixed - precision training were utilized to reduce memory usage, leading to computational speedups and optimal resource utilization. Fully sharded data parallelism (FSDP) with hybrid sharding was employed to balance inter - node memory efficiency and intra - node communication overhead within the cluster.

📄 License

• The Instella - 3B models are licensed for academic and research purposes under a ResearchRAIL license.
• The [amd/Instella - GSM8K - synthetic](https://huggingface.co/datasets/amd/Instella - GSM8K - synthetic) dataset used in second stage pre - training is built with Qwen2.5 - 72B - Instruct, and is licensed for academic and research purposes under a ResearchRAIL license. Refer to the [LICENSE](https://huggingface.co/datasets/amd/Instella - GSM8K - synthetic/blob/main/LICENSE) and [NOTICES](https://huggingface.co/datasets/amd/Instella - GSM8K - synthetic/blob/main/NOTICES) in the [amd/Instella - GSM8K - synthetic](https://huggingface.co/datasets/amd/Instella - GSM8K - synthetic) dataset card files for more information.
• Refer to the [LICENSE](https://huggingface.co/amd/Instella - 3B/blob/main/LICENSE) and [NOTICES](https://huggingface.co/amd/Instella - 3B/blob/main/NOTICES) files for more information.

Additional Resources

Hugging Face Model Cards

• Pre - trained models:
  • Instella - 3B - Stage1: [amd/Instella - 3B - Stage1](https://huggingface.co/amd/Instella - 3B - Stage1), First stage pre - training checkpoint.
  • Instella - 3B: [amd/Instella - 3B](https://huggingface.co/amd/Instella - 3B), Final pre - training checkpoint.
• Instruction - tuned models:
  • Instella - 3B - SFT: [amd/Instella - 3B - SFT](https://huggingface.co/amd/Instella - 3B - SFT), Supervised fine - tuned checkpoint.
  • Instella - 3B - Instruct: [amd/Instella - 3B - Instruct](https://huggingface.co/amd/Instella - 3B - Instruct), Final Instruction - tuned checkpoint.

Datasets

Second stage pre - training GSM8k synthetic dataset: [amd/Instella - GSM8K - synthetic](https://huggingface.co/datasets/amd/Instella - GSM8K - synthetic)

• The dataset consists of two splits: train and train_119K.
• For Instella - 3B model second stage pre - training, the train_119K split, which is a subset of the larger train split, was used.

Code

• Github: [https://github.com/AMD - AIG - AIMA/Instella](https://github.com/AMD - AIG - AIMA/Instella)

Please refer to the following blogs to get started with using these techniques on AMD GPUs:

• [PyTorch Fully Sharded Data Parallel (FSDP) on AMD GPUs with ROCm™](https://rocm.blogs.amd.com/artificial - intelligence/fsdp - training - pytorch/README.html)
• [Accelerating Large Language Models with Flash Attention on AMD GPUs](https://rocm.blogs.amd.com/artificial - intelligence/flash - attention/README.html)
• [Accelerate PyTorch Models using torch.compile on AMD GPUs with ROCm™](https://rocm.blogs.amd.com/artificial - intelligence/torch_compile/README.html)
• [Introducing the First AMD 1B Language Models: AMD OLMo](https://www.amd.com/en/developer/resources/technical - articles/introducing - the - first - amd - 1b - language - model.html)

Bias, Risks, and Limitations

⚠️ Important Note

• The models are being released for research purposes only and are not intended for use cases that require high levels of factuality, safety - critical situations, health, or medical applications, generating false information, facilitating toxic conversations.
• Model checkpoints are made accessible without any safety promises. It is crucial for users to conduct comprehensive evaluations and implement safety filtering mechanisms as per their respective use cases.
• It may be possible to prompt the model to generate content that may be factually inaccurate, harmful, violent, toxic, biased, or otherwise objectionable. Such content may also get generated by prompts that did not intend to produce output as such. Users are thus requested to be aware of this and exercise caution and responsible thinking when using the model.
• Multi - lingual abilities of the models have not been tested and thus may misunderstand and generate erroneous responses across different languages.

Citations

@misc{Instella,
    title = {Instella: Fully Open Language Models with Stellar Performance},
    url = {https://huggingface.co/amd/Instella-3B},
    author = {Jiang Liu, Jialian Wu, Xiaodong Yu, Prakamya Mishra, Sudhanshu Ranjan, Zicheng Liu, Chaitanya Manem, Yusheng Su, Pratik Prabhanjan Brahma, Gowtham Ramesh, Ximeng Sun, Ze Wang, Emad Barsoum},
    month = {March},
    year = {2025}
}