Magistral Small 2506

🚀 Magistral-Small-2506模型卡片

Magistral-Small-2506基於Mistral Small 3.1（2503）構建，新增推理能力，經過從Magistral Medium痕跡進行的SFT和後續的RL優化，是一個擁有240億參數的小型高效推理模型。

Magistral Small可以進行本地部署，量化後可適配單張RTX 4090或配備32GB內存的MacBook。

在我們的博客文章中瞭解更多關於Magistral的信息。

該模型在論文Magistral中被提出。

🚀 快速開始

Magistral-Small-2506可在本地部署，量化後能適配單張RTX 4090或32GB內存的MacBook。你可以參考以下不同使用場景的具體操作。

✨ 主要特性

• 推理能力：在給出答案之前，能夠進行長鏈推理。
• 多語言支持：支持數十種語言，包括英語、法語、德語、希臘語、印地語、印尼語、意大利語、日語、韓語、馬來語、尼泊爾語、波蘭語、葡萄牙語、羅馬尼亞語、俄語、塞爾維亞語、西班牙語、土耳其語、烏克蘭語、越南語、阿拉伯語、孟加拉語、中文和波斯語。
• Apache 2.0許可證：開放許可，允許商業和非商業用途的使用和修改。
• 上下文窗口：擁有128k的上下文窗口，但超過40k時性能可能會下降。因此，我們建議將最大模型長度設置為40k。

📊 基準測試結果

💡 採樣參數

請確保使用以下參數：

• top_p：0.95
• temperature：0.7
• max_tokens：40960

💻 使用示例

基本聊天模板

我們強烈建議在使用時包含RL期間使用的默認系統提示，以獲得最佳效果。你可以根據具體用例進行編輯和定製。

<s>[SYSTEM_PROMPT]system_prompt

A user will ask you to solve a task. You should first draft your thinking process (inner monologue) until you have derived the final answer. Afterwards, write a self-contained summary of your thoughts (i.e. your summary should be succinct but contain all the critical steps you needed to reach the conclusion). You should use Markdown to format your response. Write both your thoughts and summary in the same language as the task posed by the user. NEVER use \boxed{} in your response.

Your thinking process must follow the template below:
<think>
Your thoughts or/and draft, like working through an exercise on scratch paper. Be as casual and as long as you want until you are confident to generate a correct answer.
</think>

Here, provide a concise summary that reflects your reasoning and presents a clear final answer to the user. Don't mention that this is a summary.

Problem:

[/SYSTEM_PROMPT][INST]user_message[/INST]<think>
reasoning_traces
</think>
assistant_response</s>[INST]user_message[/INST]

system_prompt、user_message和assistant_response為佔位符。

你可以根據用例和需求，選擇在多輪交互中保留推理痕跡或僅保留最終的助手回覆。

請確保以mistral-common作為參考標準

不同框架下的使用

推理

• vllm (推薦)：詳情見下文

此外，社區還提供了該模型的量化版本，可與以下框架配合使用（按字母順序排序）：

• llama.cpp：https://huggingface.co/mistralai/Magistral-Small-2506_gguf
• lmstudio (llama.cpp, MLX)：https://lmstudio.ai/models/mistralai/magistral-small
• ollama：https://ollama.com/library/magistral
• unsloth (llama.cpp)：https://huggingface.co/unsloth/Magistral-Small-2506-GGUF

訓練

可使用以下框架進行微調（按字母順序排序）：

• axolotl：https://github.com/axolotl-ai-cloud/axolotl/tree/main/examples/magistral
• unsloth：https://docs.unsloth.ai/basics/magistral

其他

你還可以在以下平臺使用Magistral：

• kaggle：https://www.kaggle.com/models/mistral-ai/magistral-small-2506

vLLM（推薦）

我們建議使用vLLM庫來實現可用於生產環境的推理管道。

📦 安裝指南

確保安裝最新的vLLM代碼：

pip install -U vllm \
    --pre \
    --extra-index-url https://wheels.vllm.ai/nightly

這樣做應該會自動安裝mistral_common >= 1.6.0。

要進行檢查，可運行：

python -c "import mistral_common; print(mistral_common.__version__)"

你也可以使用現成的Docker鏡像或在Docker Hub上獲取。

服務模型

按以下方式啟動服務：

vllm serve mistralai/Magistral-Small-2506 --tokenizer_mode mistral --config_format mistral --load_format mistral --tool-call-parser mistral --enable-auto-tool-choice --tensor-parallel-size 2

調用模型

from openai import OpenAI
from huggingface_hub import hf_hub_download

# Modify OpenAI's API key and API base to use vLLM's API server.
openai_api_key = "EMPTY"
openai_api_base = "http://localhost:8000/v1"

TEMP = 0.7
TOP_P = 0.95
MAX_TOK = 40_960

client = OpenAI(
    api_key=openai_api_key,
    base_url=openai_api_base,
)

models = client.models.list()
model = models.data[0].id

def load_system_prompt(repo_id: str, filename: str) -> str:
    file_path = hf_hub_download(repo_id=repo_id, filename=filename)
    with open(file_path, "r") as file:
        system_prompt = file.read()
    return system_prompt

SYSTEM_PROMPT = load_system_prompt(model, "SYSTEM_PROMPT.txt")

query = "Write 4 sentences, each with at least 8 words. Now make absolutely sure that every sentence has exactly one word less than the previous sentence."
# or try out other queries
# query = "Exactly how many days ago did the French Revolution start? Today is June 4th, 2025."
# query = "Think about 5 random numbers. Verify if you can combine them with addition, multiplication, subtraction or division to 133"
# query = "If it takes 30 minutes to dry 12 T-shirts in the sun, how long does it take to dry 33 T-shirts?"

messages = [
    {"role": "system", "content": SYSTEM_PROMPT},
    {"role": "user", "content": query}
]
stream = client.chat.completions.create(
  model=model,
  messages=messages,
  stream=True,
  temperature=TEMP,
  top_p=TOP_P,
  max_tokens=MAX_TOK,
)

print("client: Start streaming chat completions...")
printed_content = False

for chunk in stream:
  content = None
  # Check the content is content
  if hasattr(chunk.choices[0].delta, "content"):
    content = chunk.choices[0].delta.content

  if content is not None:
    if not printed_content:
        printed_content = True
        print("\ncontent:", end="", flush=True)
    # Extract and print the content
    print(content, end="", flush=True)

# content:<think>
# Alright, I need to write 4 sentences where each one has at least 8 words and each subsequent sentence has one fewer word than the previous one.
# ...
# Final boxed answer (the four sentences):

# \[
# \boxed{
# \begin{aligned}
# &\text{1. The quick brown fox jumps over lazy dog and yells hello.} \\
# &\text{2. I saw the cat on the stair with my hat.} \\
# &\text{3. The man in the moon came down quickly today.} \\
# &\text{4. A cat sat on the mat today patiently.}
# \end{aligned}
# }
# \]

📚 詳細文檔

模型信息

額外提示

⚠️ 重要提示

如果你想了解我們如何處理你的個人數據，請閱讀我們的隱私政策。

📄 許可證

本模型採用Apache 2.0許可證，允許商業和非商業用途的使用和修改。