Mistral Small 24B Instruct 2501 Reasoning

🚀 Mistral-Small-Reasoning

This is an instruction-tuned language model for reasoning, fine-tuned from Mistral-Small-24B-Instruct-2501. It's optimized for mathematical reasoning tasks and has shown good performance on multiple datasets.

🚀 Quick Start

A demo is available at twllm.com, and inference can be run using vLLM or sglang.

✨ Features

• Instruction-tuned: Specifically optimized for mathematical reasoning tasks.
• Fine-tuned on multiple datasets: Including OpenR1-Math-220k and s1K-1.1.
• Good performance: Achieved high pass@1 scores on various datasets such as MATH-500, AIME 2025, etc.

📚 Documentation

📦 Model Details

📋 Training Details

The model was trained using 4×8 H100 GPUs, provided by Ubitus.

base_model: mistralai/Mistral-Small-24B-Instruct-2501

plugins:
  - axolotl.integrations.liger.LigerPlugin
liger_rope: true
liger_rms_norm: true
liger_swiglu: true
liger_fused_linear_cross_entropy: true

datasets:
  - path: yentinglin/s1K-1.1-trl-format
    type: chat_template
    chat_template: tokenizer_default
    field_messages: messages
    message_field_role: role
    message_field_content: content
  - path: open-r1/OpenR1-Math-220k
    type: chat_template
    chat_template: tokenizer_default
    field_messages: messages
    message_field_role: from
    message_field_content: value
dataset_prepared_path:
val_set_size: 0.0
output_dir: ./placeholder/

sequence_len: 32768
sample_packing: true
eval_sample_packing: False
pad_to_sequence_len: true

wandb_project: Reasoning
wandb_entity:
wandb_watch:
wandb_name: Mistral-24B-SFT-220k
wandb_log_model:

gradient_accumulation_steps: 4
micro_batch_size: 1
num_epochs: 5
optimizer: adamw_torch_fused
lr_scheduler: cosine
learning_rate: 2e-5

train_on_inputs: false
group_by_length: false
bf16: auto
tf32: false

gradient_checkpointing: true
gradient_checkpointing_kwargs:
  use_reentrant: false
logging_steps: 1
flash_attention: true

warmup_ratio: 0.1
saves_per_epoch: 2
weight_decay: 0.0
deepspeed: deepspeed_configs/zero3_bf16.json
special_tokens:
  pad_token: "<pad>"

🔍 Evaluation

The evaluation code is available at Hugging Face Open-R1. Note that I have updated the AIME 25 dataset to the full set, available at AIME 2025.

Our results below are averaged over multiple runs. See our eval details here.

📄 License

The model is licensed under Apache 2.0.

📖 Citation

If you use this model, please cite:

@article{yentinglin2025_mistral_reasoning,
  author = {Yenting Lin},
  title = {Mistral-Small-24B-Instruct-2501-reasoning},
  journal = {Hugging Face},
  year = {2025},
  url = {https://huggingface.co/yentinglin/Mistral-Small-24B-Instruct-2501-reasoning}
}

⚠️ Disclaimer

This model is provided “as‑is” and without warranties of any kind. Users are solely responsible for evaluating the accuracy and suitability of the outputs. The developers assume no liability for any direct or indirect damages arising from its use.
The model is strictly not intended for high‑risk applications such as medical diagnosis, legal advice, or financial investment. For such use cases, please consult qualified professionals.