Mistral Small 3.1 24B Instruct 2503 GGUF

🚀 Mistral-Small-3.1-24B-Instruct-2503 GGUF模型

Mistral-Small-3.1-24B-Instruct-2503 GGUF模型是基於Mistral架構的高性能語言模型，具備多語言支持、視覺理解等先進能力，適用於多種自然語言處理和視覺分析任務。

🚀 快速開始

安裝依賴

若要使用該模型，推薦使用vLLM庫來實現生產就緒的推理管道。確保安裝vLLM >= 0.8.1：

pip install vllm --upgrade

這將自動安裝mistral_common >= 1.5.4。你可以通過以下命令進行檢查：

python -c "import mistral_common; print(mistral_common.__version__)"

你也可以使用現成的Docker鏡像或在Docker Hub上獲取。

啟動服務

建議在服務器/客戶端環境中使用Mistral-Small-3.1-24B-Instruct-2503：

1. 啟動服務器：

vllm serve mistralai/Mistral-Small-3.1-24B-Instruct-2503 --tokenizer_mode mistral --config_format mistral --load_format mistral --tool-call-parser mistral --enable-auto-tool-choice --limit_mm_per_prompt 'image=10' --tensor-parallel-size 2

注意：在GPU上運行Mistral-Small-3.1-24B-Instruct-2503需要約55GB的GPU顯存（bf16或fp16）。

2. 使用Python代碼向客戶端發送請求：

import requests
import json
from huggingface_hub import hf_hub_download
from datetime import datetime, timedelta

url = "http://<your-server-url>:8000/v1/chat/completions"
headers = {"Content-Type": "application/json", "Authorization": "Bearer token"}

model = "mistralai/Mistral-Small-3.1-24B-Instruct-2503"


def load_system_prompt(repo_id: str, filename: str) -> str:
    file_path = hf_hub_download(repo_id=repo_id, filename=filename)
    with open(file_path, "r") as file:
        system_prompt = file.read()
    today = datetime.today().strftime("%Y-%m-%d")
    yesterday = (datetime.today() - timedelta(days=1)).strftime("%Y-%m-%d")
    model_name = repo_id.split("/")[-1]
    return system_prompt.format(name=model_name, today=today, yesterday=yesterday)


SYSTEM_PROMPT = load_system_prompt(model, "SYSTEM_PROMPT.txt")

image_url = "https://huggingface.co/datasets/patrickvonplaten/random_img/resolve/main/europe.png"

messages = [
    {"role": "system", "content": SYSTEM_PROMPT},
    {
        "role": "user",
        "content": [
            {
                "type": "text",
                "text": "Which of the depicted countries has the best food? Which the second and third and fourth? Name the country, its color on the map and one its city that is visible on the map, but is not the capital. Make absolutely sure to only name a city that can be seen on the map.",
            },
            {"type": "image_url", "image_url": {"url": image_url}},
        ],
    },
]

data = {"model": model, "messages": messages, "temperature": 0.15}

response = requests.post(url, headers=headers, data=json.dumps(data))
print(response.json()["choices"][0]["message"]["content"])
# Determining the "best" food is highly subjective and depends on personal preferences. However, based on general popularity and recognition, here are some countries known for their cuisine:

# 1. **Italy** - Color: Light Green - City: Milan
#    - Italian cuisine is renowned worldwide for its pasta, pizza, and various regional specialties.

# 2. **France** - Color: Brown - City: Lyon
#    - French cuisine is celebrated for its sophistication, including dishes like coq au vin, bouillabaisse, and pastries like croissants and éclairs.

# 3. **Spain** - Color: Yellow - City: Bilbao
#    - Spanish cuisine offers a variety of flavors, from paella and tapas to jamón ibérico and churros.

# 4. **Greece** - Not visible on the map
#    - Greek cuisine is known for dishes like moussaka, souvlaki, and baklava. Unfortunately, Greece is not visible on the provided map, so I cannot name a city.

# Since Greece is not visible on the map, I'll replace it with another country known for its good food:

# 4. **Turkey** - Color: Light Green (east part of the map) - City: Istanbul
#    - Turkish cuisine is diverse and includes dishes like kebabs, meze, and baklava.

✨ 主要特性

• 視覺理解：模型具備視覺能力，可分析圖像並基於視覺內容提供見解。
• 多語言支持：支持數十種語言，包括英語、法語、德語、希臘語、印地語、印尼語、意大利語、日語、韓語、馬來語、尼泊爾語、波蘭語、葡萄牙語、羅馬尼亞語、俄語、塞爾維亞語、西班牙語、瑞典語、土耳其語、烏克蘭語、越南語、阿拉伯語、孟加拉語、中文、波斯語。
• 以代理為中心：提供一流的代理能力，支持原生函數調用和JSON輸出。
• 高級推理：具備先進的對話和推理能力。
• 開源許可：採用Apache 2.0許可證，允許商業和非商業用途的使用和修改。
• 長上下文窗口：擁有128k的上下文窗口。
• 系統提示支持：對系統提示有強大的遵循和支持能力。
• 分詞器：使用Tekken分詞器，詞彙表大小為131k。

📦 安裝指南

vLLM安裝

確保安裝vLLM >= 0.8.1：

pip install vllm --upgrade

安裝後，可通過以下命令檢查mistral_common版本：

python -c "import mistral_common; print(mistral_common.__version__)"

也可使用Docker鏡像或從Docker Hub獲取。

💻 使用示例

基礎用法

import requests
import json
from huggingface_hub import hf_hub_download
from datetime import datetime, timedelta

url = "http://<your-server-url>:8000/v1/chat/completions"
headers = {"Content-Type": "application/json", "Authorization": "Bearer token"}

model = "mistralai/Mistral-Small-3.1-24B-Instruct-2503"


def load_system_prompt(repo_id: str, filename: str) -> str:
    file_path = hf_hub_download(repo_id=repo_id, filename=filename)
    with open(file_path, "r") as file:
        system_prompt = file.read()
    today = datetime.today().strftime("%Y-%m-%d")
    yesterday = (datetime.today() - timedelta(days=1)).strftime("%Y-%m-%d")
    model_name = repo_id.split("/")[-1]
    return system_prompt.format(name=model_name, today=today, yesterday=yesterday)


SYSTEM_PROMPT = load_system_prompt(model, "SYSTEM_PROMPT.txt")

image_url = "https://huggingface.co/datasets/patrickvonplaten/random_img/resolve/main/europe.png"

messages = [
    {"role": "system", "content": SYSTEM_PROMPT},
    {
        "role": "user",
        "content": [
            {
                "type": "text",
                "text": "Which of the depicted countries has the best food? Which the second and third and fourth? Name the country, its color on the map and one its city that is visible on the map, but is not the capital. Make absolutely sure to only name a city that can be seen on the map.",
            },
            {"type": "image_url", "image_url": {"url": image_url}},
        ],
    },
]

data = {"model": model, "messages": messages, "temperature": 0.15}

response = requests.post(url, headers=headers, data=json.dumps(data))
print(response.json()["choices"][0]["message"]["content"])

高級用法

import requests
import json
from huggingface_hub import hf_hub_download
from datetime import datetime, timedelta

url = "http://<your-url>:8000/v1/chat/completions"
headers = {"Content-Type": "application/json", "Authorization": "Bearer token"}

model = "mistralai/Mistral-Small-3.1-24B-Instruct-2503"


def load_system_prompt(repo_id: str, filename: str) -> str:
    file_path = hf_hub_download(repo_id=repo_id, filename=filename)
    with open(file_path, "r") as file:
        system_prompt = file.read()
    today = datetime.today().strftime("%Y-%m-%d")
    yesterday = (datetime.today() - timedelta(days=1)).strftime("%Y-%m-%d")
    model_name = repo_id.split("/")[-1]
    return system_prompt.format(name=model_name, today=today, yesterday=yesterday)


SYSTEM_PROMPT = load_system_prompt(model, "SYSTEM_PROMPT.txt")


tools = [
    {
        "type": "function",
        "function": {
            "name": "get_current_weather",
            "description": "Get the current weather in a given location",
            "parameters": {
                "type": "object",
                "properties": {
                    "city": {
                        "type": "string",
                        "description": "The city to find the weather for, e.g. 'San Francisco'",
                    },
                    "state": {
                        "type": "string",
                        "description": "The state abbreviation, e.g. 'CA' for California",
                    },
                    "unit": {
                        "type": "string",
                        "description": "The unit for temperature",
                        "enum": ["celsius", "fahrenheit"],
                    },
                },
                "required": ["city", "state", "unit"],
            },
        },
    },
    {
        "type": "function",
        "function": {
            "name": "rewrite",
            "description": "Rewrite a given text for improved clarity",
            "parameters": {
                "type": "object",
                "properties": {
                    "text": {
                        "type": "string",
                        "description": "The input text to rewrite",
                    }
                },
            },
        },
    },
]

messages = [
    {"role": "system", "content": SYSTEM_PROMPT},
    {
        "role": "user",
        "content": "Could you please make the below article more concise?\n\nOpenAI is an artificial intelligence research laboratory consisting of the non-profit OpenAI Incorporated and its for-profit subsidiary corporation OpenAI Limited Partnership.",
    },
    {
        "role": "assistant",
        "content": "",
        "tool_calls": [
            {
                "id": "bbc5b7ede",
                "type": "function",
                "function": {
                    "name": "rewrite",
                    "arguments": '{"text": "OpenAI is an artificial intelligence research laboratory consisting of the non-profit OpenAI Incorporated and its for-profit subsidiary corporation OpenAI Limited Partnership."}',
                },
            }
        ],
    },
    {
        "role": "tool",
        "content": '{"action":"rewrite","outcome":"OpenAI is a FOR-profit company."}',
        "tool_call_id": "bbc5b7ede",
        "name": "rewrite",
    },
    {
        "role": "assistant",
        "content": "---\n\nOpenAI is a FOR-profit company.",
    },
    {
        "role": "user",
        "content": "Can you tell me what the temperature will be in Dallas, in Fahrenheit?",
    },
]

data = {"model": model, "messages": messages, "tools": tools, "temperature": 0.15}

response = requests.post(url, headers=headers, data=json.dumps(data))
print(response.json()["choices"][0]["message"]["tool_calls"])

📚 詳細文檔

模型生成細節

該模型使用llama.cpp在提交版本92ecdcc0時生成。

超低比特量化（1 - 2比特）

最新的量化方法為超低比特模型（1 - 2比特）引入了精度自適應量化，在Llama-3-8B上經基準測試證明有顯著改進。該方法採用特定層策略，在保持極高內存效率的同時保留準確性。

基準測試上下文

所有測試均在Llama-3-8B-Instruct上進行，使用標準困惑度評估管道、2048令牌上下文窗口和相同的提示集。

量化性能比較（Llama-3-8B）

模型格式選擇

選擇正確的模型格式取決於硬件能力和內存限制：

包含文件及詳情

測試說明

若發現這些模型有用，請點擊“點贊”。可幫助測試AI網絡監控助手，選擇AI助手類型：

• TurboLLM (GPT-4o-mini)
• HugLLM (Hugginface開源)
• TestLLM (僅支持CPU的實驗性模型)

測試內容

正在測試小型開源模型在AI網絡監控中的極限，包括函數調用、模型大小與任務處理能力等。

示例命令

1. "Give me info on my websites SSL certificate"
2. "Check if my server is using quantum safe encyption for communication"
3. "Run a comprehensive security audit on my server"
4. "Create a cmd processor to .. (what ever you want)"

🔧 技術細節

基礎指令模板（V7-Tekken）

<s>[SYSTEM_PROMPT]<system prompt>[/SYSTEM_PROMPT][INST]<user message>[/INST]<assistant response></s>[INST]<user message>[/INST]

<system_prompt>、<user message>和<assistant response>為佔位符。請確保使用mistral-common作為參考。

基準測試結果

預訓練評估

指令評估 - 文本

指令評估 - 視覺

多語言評估

長上下文評估

📄 許可證

該模型採用Apache 2.0許可證。

⚠️ 重要提示

建議使用相對較低的溫度，如temperature=0.15。確保為模型添加系統提示以滿足特定需求。

💡 使用建議

若使用Transformers庫，其實現未經過充分測試，僅進行了“氛圍檢查”，因此使用vLLM庫能確保100%正確的行為。