Mimo 7B SFT

🚀 MiMo-7B: Unlocking the Reasoning Potential of Language Model

MiMo-7B is a series of models trained from scratch, born for reasoning tasks. It unlocks the reasoning potential of language models through both pre - training and post - training strategies, showing excellent performance in mathematics and code reasoning tasks.

🚀 Quick Start

To quickly start using the MiMo - 7B series models, you can choose different inference methods according to your needs. The following are examples of different inference methods:

SGLang Inference

# Install the latest SGlang from main branch
python3 -m uv pip install "sglang[all] @ git+https://github.com/sgl-project/sglang.git/@main#egg=sglang&subdirectory=python"

# Launch SGLang Server
python3 -m sglang.launch_server --model-path XiaomiMiMo/MiMo-7B-SFT --host 0.0.0.0 --trust-remote-code

Detailed usage can be found in SGLang documents. MTP will also be supported in 24h.

vLLM inference

Recommended method

from vllm import LLM, SamplingParams

model_path = "/path/to/MiMo"
llm = LLM(
    model=model_path,
    trust_remote_code=True,
    num_speculative_tokens=1,
    disable_log_stats=False
)
sampling_params = SamplingParams(temperature=0.6)

conversation = [
    {
        "role": "system",
        "content": ""
    },
    {
        "role": "user",
        "content": "Write an essay about the importance of higher education.",
    },
]

outputs = llm.chat(conversation,
                   sampling_params=sampling_params,
                   use_tqdm=False)

for output in outputs:
    prompt = output.prompt
    generated_text = output.outputs[0].text
    print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")

print("=" * 80)

Alternative method

import register_mimo_in_vllm

from vllm import LLM, SamplingParams

model_path = "/path/to/MiMo"
llm = LLM(
    model=model_path,
    trust_remote_code=True,
    # num_speculative_tokens=1,
    disable_log_stats=False
)
sampling_params = SamplingParams(temperature=0.6)

HuggingFace inference

from transformers import AutoModel, AutoModelForCausalLM, AutoTokenizer

model_id = "XiaomiMiMo/MiMo-7B-SFT"
model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained(model_id)
inputs = tokenizer(["Today is"], return_tensors='pt')
output = model.generate(**inputs)

✨ Features

Pre - Training: Base Model Born for Reasoning

• Optimize the data pre - processing pipeline, enhance text extraction toolkits, and apply multi - dimensional data filtering to increase the reasoning pattern density in pre - training data. Also, use multiple strategies to generate massive diverse synthetic reasoning data.
• Adopt a three - stage data mixture strategy for pre - training. MiMo - 7B - Base is pre - trained on approximately 25 trillion tokens.
• Incorporate Multiple - Token Prediction as an additional training objective to enhance model performance and accelerate inference.

Post - Training Recipe: Pioneering Reasoning Model

• Curate 130K mathematics and code problems as RL training data, which can be verified by rule - based verifiers. Each problem undergoes careful cleaning and difficulty assessment to ensure quality. Only rule - based accuracy rewards are employed to avoid potential reward hacking.
• Introduce a test difficulty driven code reward to mitigate the sparse reward issue for challenging code problems. By assigning fine - grained scores for test cases with varying difficulty levels, the policy can be more effectively optimized via dense reward signal.
• Implement a data re - sampling strategy for easy problems to enhance rollout sampling efficiency and stabilize policy updates, especially in the later phases of RL training.

RL Infrastructure

• Develop a Seamless Rollout Engine to accelerate RL training and validation. The design integrates continuous rollout, asynchronous reward computation, and early termination to minimize GPU idle time, achieving $2.29\times$ faster training and $1.96\times$ faster validation.
• Support MTP in vLLM and enhance the robustness of the inference engine in the RL system.

📦 Installation

The installation steps are included in the inference examples above. For SGLang, you need to install it from the main branch; for vLLM, you can use the official fork or register a loader; for HuggingFace, you need to install the transformers library.

💻 Usage Examples

The usage examples are provided in the "Quick Start" section, including code examples for SGLang, vLLM, and HuggingFace inference.

📚 Documentation

• The detailed usage of SGLang can be found in SGLang documents.
• The technical report can be found at https://arxiv.org/abs/2505.07608.
• The models are available on HuggingFace and ModelScope.

🔧 Technical Details

• The MTP layers of MiMo - 7B are tuned during pretraining and SFT and freezed during RL. With one MTP layer for speculative decoding, the acceptance rate is about 90%.
• The RL experiments from MiMo - 7B - Base show that the model has extraordinary reasoning potential, even surpassing much larger 32B models. The MiMo - 7B - RL model, obtained by performing RL training on a cold - started SFT model, demonstrates superior performance on both mathematics and code reasoning tasks, matching the performance of OpenAI o1 - mini.

📄 License

This model repository is licensed under the MIT License.

Model Information

Evaluation Results

General Benchmarks

Mathematics Benchmarks

Code Benchmarks

MiMo - 7B Series Benchmarks

⚠️ Important Note

The evaluations are conducted with temperature = 0.6.

AIME24 and AIME25 are with averaged score of 32 repetitions. LiveCodeBench v5 (20240801 - 20250201), LiveCodeBench v6 (20250201 - 20250501), GPQA - Diamond and IF - Eval are with averaged score of 8 repetitions. MATH500 and SuperGPQA are with a single run.