Arcana Qwen3 2.4B A0.6B

🚀 Arcana Qwen3 2.4B A0.6B

This is a MoE (Mixture of Experts) Qwen3 model aiming to provide more accurate results with higher efficiency and less memory usage.

"We are all experts at something, but we’re all also beginners at something else."

— The Imitation Game (2014)

🚀 Quick Start

This is a MoE (Mixture of Experts) Qwen3 model which has a total of 2.4B parameters and 0.6B for each of the 4 experts. All the expert models are detailed below.

✨ Features

Model Information

Expert Models

Key Training Parameters (SFTConfig)

• per_device_train_batch_size = 2
• gradient_accumulation_steps = 4
• warmup_steps = 5
• num_train_epochs = 1
• learning_rate = 2e-5
• optim = "adamw_8bit"
• weight_decay = 0.01
• seed = 3407

Coding

suayptalha/Qwen3-0.6B-Code-Expert

This model was fully fine - tuned with BF16 on the first 20k rows of the nvidia/OpenCodeReasoning dataset for 1 epoch.

Math

suayptalha/Qwen3-0.6B-Math-Expert

This model was fully fine - tuned with BF16 on the entire unsloth/OpenMathReasoning - mini dataset for 1 epoch.

Medical

suayptalha/Qwen3-0.6B-Medical-Expert

This model was fully fine - tuned with BF16 on the first 20k rows of the FreedomIntelligence/medical - o1 - reasoning - SFT dataset for 1 epoch.

Instruction Following

Qwen/Qwen3-0.6B

The Qwen/Qwen3-0.6B model was directly used for this expert without fine - tuning.

Router Model

The router model can be found here, which is a trained version of distilbert/distilbert - base - uncased on 7 different datasets.

💻 Usage Examples

Basic Usage

import torch
from huggingface_hub import snapshot_download
from transformers import AutoModelForCausalLM, AutoTokenizer

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

local_dir = snapshot_download(
    repo_id="suayptalha/Qwen3-2.4B-A0.6B",
)

model = AutoModelForCausalLM.from_pretrained(
    local_dir,
    trust_remote_code=True,
)
tokenizer = AutoTokenizer.from_pretrained(
    local_dir,
)

model.to(device)
model.eval()

prompt = "I have pain in my chest, what should I do?"
messages = [{"role": "user", "content": prompt}]

prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)

with torch.no_grad():
    output_ids = model.generate(
        text=prompt,
        max_new_tokens=1024,
        temperature=0.6,
        top_p=0.95,
    )
output_text = tokenizer.decode(output_ids[0], skip_special_tokens=True)
print(output_text)

📄 License

This project is licensed under the Apache License 2.0. See the LICENSE file for details.

Support