Motif 2.6B

🚀 Motif 2.6B

Motif 2.6B is a 2.6 billion parameter language model trained from scratch. It aims to match the performance of well - known open - source models and is our first step towards building AI aligned with human values.

🚀 Quick Start

We are excited to announce Motif 2.6B, a language model with 2.6 billion parameters. It has been trained from scratch on AMD Instinct™ MI250 GPUs. This release represents our initial foray into creating a helpful and reliable AI that aligns with human values. Our goal with Motif 2.6B is to achieve performance comparable to well - known open - source models such as Gemma, Llama, and Phi, especially those in the sLLM regime.

📦 Installation

No installation steps are provided in the original document, so this section is skipped.

✨ Features

Training information

• GPUs: 384 MI250
• Training time: 42 days
• Training data: 2.4T tokens

Notice: A detailed technical report will be released at a later time.

Evaluation

When models are released, their developers often present benchmark results based on their chosen evaluation settings in technical reports or papers. This common practice can lead to difficulties in comparing models across different organizations, as evaluation scores can vary depending on conditions that are not always fully disclosed. To address this, we reference performance scores from each model’s official publications.

In the Evaluation Appendix, we provide specific examples of benchmark score differences for major models to illustrate the variability of evaluation scores across reports.

Comparison to Mistral 7B by Mistral AI

The benchmarks and scores in the table below are from the Mistral 7B technical report.

* : We report the 4 - shot, maj@1 score instead of the 4 - shot, maj@4.

Comparison to the Gemma series by Google

Gemma 1 & 2

The benchmarks and scores in the table below are from the Gemma 2 technical report.

Note: Although referred to as "2B", Gemma 2 2B actually has 2.6 billion parameters.

Gemma 3

The benchmarks and scores in the table below are from the Gemma 3 technical report.

Comparison to the Llama series by Meta

Llama 3

The benchmarks and scores in the table below are from the Llama 3 technical report.

Llama 3.2

The benchmarks and scores in the table below are from the Llama 3.2 official blog.

Comparison to the Phi series by Microsoft

The benchmarks and scores in the table below are from the Phi - 3 technical report.

📚 Evaluation Appendix

In the above comparisons, based on the benchmark scores in the original technical reports, Motif 2.6B shows average performance improvements of -15.36% and -13.45% over Llama 3 8B and Gemma 2 9B respectively. However, when compared to the benchmarks and scores in the Qwen 2.5 technical report, Motif 2.6B shows an average improvement of +19.27% over Llama 3 8B and +1.68% over Gemma 2 9B. See the table below for details.

Comparison to Llama 3 8B and Gemma 2 9B based on scores from the Qwen2.5 technical report

The benchmarks and scores in the table below are from the Qwen2.5 technical report.

💻 Usage Examples

Basic Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "Motif-Technologies/Motif-2.6B",
    trust_remote_code = True, 
    _attn_implementation = "eager", # also supports flash_attention_2
).cuda()

tokenizer = AutoTokenizer.from_pretrained(
    "Motif-Technologies/Motif-2.6B", 
    trust_remote_code = True, 
)

query = "What is the capital city of South Korea?"
input_ids = tokenizer.apply_chat_template(
    [
        {'role': 'system', 'content': 'you are an helpful assistant'},
        {'role': 'user', 'content': query},
    ],
    add_generation_prompt = True,
    return_tensors='pt',
).cuda()

output = model.generate(input_ids, max_new_tokens=128, pad_token_id=tokenizer.eos_token_id)
output = tokenizer.decode(output[0, input_ids.shape[-1]:], skip_special_tokens = True)
print(output)

"""
The capital city of South Korea is Seoul. Located in the southern part of the country, Seoul is not only the largest city in South Korea but also one of the largest metropolitan areas in the world.
It is a vibrant and dynamic city known for its rich history, cultural heritage, and modern amenities. Seoul is a major economic, cultural, and political center in East Asia, and it plays a crucial role in the region's politics, economy, and culture.
The city is divided into different administrative districts, each with its own unique characteristics and attractions.
"""

📄 License

• License: other
• License Name: motif - license
• License Link: LICENSE