Devstral Small 2505 Unsloth Bnb 4bit

🚀 Run & Fine-tune Devstral with Unsloth!

Devstral is an agentic LLM for software engineering tasks. It offers agentic coding, is lightweight, and has a long context window. Ideal for software engineering agents and local deployment.

📦 Installation

This section doesn't have specific installation steps for the project as a whole. But for using different deployment methods, here are relevant details:

• vLLM Installation:

  pip install vllm --upgrade
  

  To check the installation of mistral_common which should be installed automatically:

  python -c "import mistral_common; print(mistral_common.__version__)"
  

• Download weights for LMStudio:

  pip install -U "huggingface_hub[cli]"
  huggingface-cli download \
  "mistralai/Devstral-Small-2505_gguf" \
  --include "devstralQ4_K_M.gguf" \
  --local-dir "mistralai/Devstral-Small-2505_gguf/"
  

✨ Features

• Agentic coding: Devstral is designed to excel at agentic coding tasks, making it a great choice for software engineering agents.
• lightweight: with its compact size of just 24 billion parameters, Devstral is light enough to run on a single RTX 4090 or a Mac with 32GB RAM, making it an appropriate model for local deployment and on - device use.
• Apache 2.0 License: Open license allowing usage and modification for both commercial and non - commercial purposes.
• Context Window: A 128k context window.
• Tokenizer: Utilizes a Tekken tokenizer with a 131k vocabulary size.

💻 Usage Examples

Basic Usage

There isn't a very basic single - line usage example. But here are some common usage scenarios:

• Running with vLLM:

  vllm serve mistralai/Devstral-Small-2505 --tokenizer_mode mistral --config_format mistral --load_format mistral --tool-call-parser mistral --enable-auto-tool-choice --tensor-parallel-size 2
  

• Using OpenHands with vLLM server:
  • First, start the vLLM server as above.
  • Then, pull and run the OpenHands Docker image:

  docker pull docker.all-hands.dev/all-hands-ai/runtime:0.38-nikolaik
  docker run -it --rm --pull=always \
      -e SANDBOX_RUNTIME_CONTAINER_IMAGE=docker.all-hands.dev/all-hands-ai/runtime:0.38-nikolaik \
      -e LOG_ALL_EVENTS=true \
      -v /var/run/docker.sock:/var/run/docker.sock \
      -v ~/.openhands-state:/.openhands-state \
      -p 3000:3000 \
      --add-host host.docker.internal:host-gateway \
      --name openhands-app \
      docker.all-hands.dev/all-hands-ai/openhands:0.38
  

  • Connect to the server in OpenHands UI:
    • Custom Model: openai/mistralai/Devstral-Small-2505
    • Base URL: http://<your-server-url>:8000/v1
    • API Key: token (or any other token used to launch the server if any)

Advanced Usage

• Building a To - Do list app with Devstral and OpenHands:
  1. Prompt Devstral:

     Build a To-Do list app with the following requirements:
     - Built using FastAPI and React.
     - Make it a one page app that:
        - Allows to add a task.
        - Allows to delete a task.
        - Allows to mark a task as done.
        - Displays the list of tasks.
     - Store the tasks in a SQLite database.
     

  2. Explore the result and iterate on the app as needed, such as asking the agent to improve UX or add features.

📚 Documentation

• General Information: Devstral is an agentic LLM for software engineering tasks built under a collaboration between Mistral AI and All Hands AI. It is finetuned from Mistral-Small-3.1, having a long context window of up to 128k tokens. As a coding agent, it is text - only and the vision encoder was removed before fine - tuning. For enterprises requiring specialized capabilities, commercial models will be released. Learn more about Devstral in our blog post.
• Benchmark Results: SWE - Bench Devstral achieves a score of 46.8% on SWE - Bench Verified, outperforming prior open - source SoTA by 6%.

  Property	Details
  Model Type	Devstral (OpenHands Scaffold), GPT - 4.1 - mini (OpenAI Scaffold), Claude 3.5 Haiku (Anthropic Scaffold), SWE - smith - LM 32B (SWE - agent Scaffold)
  SWE - Bench Verified (%)	Devstral: 46.8, GPT - 4.1 - mini: 23.6, Claude 3.5 Haiku: 40.6, SWE - smith - LM 32B: 40.2

🔧 Technical Details

• Devstral is based on Mistral-Small-3.1 and has a long context window of up to 128k tokens.
• It uses a Tekken tokenizer with a 131k vocabulary size.
• As a coding agent, it is text - only after the vision encoder was removed before fine - tuning from Mistral-Small-3.1.

📄 License

This project is licensed under the Apache 2.0 license.

Other Information

• Supported Languages: en, fr, de, es, pt, it, ja, ko, ru, zh, ar, fa, id, ms, ne, pl, ro, sr, sv, tr, uk, vi, hi, bn
• Library Name: vllm
• Inference: false
• Base Model: mistralai/Devstral-Small-2505
• Pipeline Tag: text2text - generation
• Tags: unsloth

⚠️ Important Note

Learn to run Devstral correctly - Read our Guide. Unsloth Dynamic 2.0 achieves superior accuracy & outperforms other leading quants.

💡 Usage Tip

For local deployment, Devstral's compact size makes it suitable to run on a single RTX 4090 or a Mac with 32GB RAM. For enterprises, commercial models with specialized capabilities will be released.