Devstral Small 2505 GGUF

🚀 Devstral-Small-2505 GGUF Models

A set of models for software engineering tasks with advanced quantization and various deployment options.

🚀 Quick Start

This README provides details about the Devstral-Small-2505 GGUF models, including their generation, quantization methods, model format selection, and usage instructions.

✨ Features

Model Generation

• Generated using llama.cpp at commit f5cd27b7.

Ultra-Low-Bit Quantization

• Introduces precision-adaptive quantization for ultra-low-bit models (1 - 2 bit).
• Uses layer-specific strategies to preserve accuracy and maintain extreme memory efficiency.
• Benchmark-proven improvements on Llama-3-8B.

Model Format Selection

• Offers multiple model formats (BF16, F16, Quantized Models, Very Low-Bit Quantization) to suit different hardware capabilities and memory constraints.

Agentic Coding

• Designed for agentic coding tasks, making it suitable for software engineering agents.

Long Context Window

• Has a 128k context window, allowing for more extensive input processing.

Open License

• Licensed under Apache 2.0, enabling both commercial and non-commercial use.

📦 Installation

Prerequisites

• Ensure you have the necessary hardware and software requirements for the chosen model format.
• For API usage, create a Mistral account and obtain an API key.

Deployment

• API: Follow these instructions to create an account and get an API key. Then run the provided Docker commands to start the OpenHands docker container.
• Local Inference: Use LMStudio or other providers like vllm, mistral-inference, transformers, or ollama. Follow the respective instructions for each provider.
• OpenHands: Launch an OpenAI-compatible server (e.g., vLLM or Ollama) and then use OpenHands to interact with the model. Follow the detailed steps for launching OpenHands and connecting to the server.

💻 Usage Examples

API Usage

export MISTRAL_API_KEY=<MY_KEY>

docker pull docker.all-hands.dev/all-hands-ai/runtime:0.39-nikolaik

mkdir -p ~/.openhands-state && echo '{"language":"en","agent":"CodeActAgent","max_iterations":null,"security_analyzer":null,"confirmation_mode":false,"llm_model":"mistral/devstral-small-2505","llm_api_key":"'$MISTRAL_API_KEY'","remote_runtime_resource_factor":null,"github_token":null,"enable_default_condenser":true}' > ~/.openhands-state/settings.json

docker run -it --rm --pull=always \
    -e SANDBOX_RUNTIME_CONTAINER_IMAGE=docker.all-hands.dev/all-hands-ai/runtime:0.39-nikolaik \
    -e LOG_ALL_EVENTS=true \
    -v /var/run/docker.sock:/var/run/docker.sock \
    -v ~/.openhands-state:/.openhands-state \
    -p 3000:3000 \
    --add-host host.docker.internal:host-gateway \
    --name openhands-app \
    docker.all-hands.dev/all-hands-ai/openhands:0.39

Local Inference

docker pull docker.all-hands.dev/all-hands-ai/runtime:0.38-nikolaik
docker run -it --rm --pull=always \
    -e SANDBOX_RUNTIME_CONTAINER_IMAGE=docker.all-hands.dev/all-hands-ai/runtime:0.38-nikolaik \
    -e LOG_ALL_EVENTS=true \
    -v /var/run/docker.sock:/var/run/docker.sock \
    -v ~/.openhands-state:/.openhands-state \
    -p 3000:3000 \
    --add-host host.docker.internal:host-gateway \
    --name openhands-app \
    docker.all-hands.dev/all-hands-ai/openhands:0.38

OpenHands Usage

vllm serve mistralai/Devstral-Small-2505 --tokenizer_mode mistral --config_format mistral --load_format mistral --tool-call-parser mistral --enable-auto-tool-choice --tensor-parallel-size 2

📚 Documentation

Ultra-Low-Bit Quantization

• Precision-Adaptive Quantization: Our latest method uses layer-specific strategies for ultra-low-bit models (1 - 2 bit), with proven improvements on Llama-3-8B.

• Benchmark Context: All tests were conducted on Llama-3-8B-Instruct using a standard perplexity evaluation pipeline, a 2048-token context window, and the same prompt set across all quantizations.

• Method:

  • Dynamic Precision Allocation: First/Last 25% of layers use IQ4_XS (selected layers), and the Middle 50% use IQ2_XXS/IQ3_S to increase efficiency.
  • Critical Component Protection: Embeddings/output layers use Q5_K, reducing error propagation by 38% compared to standard 1 - 2 bit.

• Quantization Performance Comparison (Llama-3-8B): | Quantization | Standard PPL | DynamicGate PPL | ∆ PPL | Std Size | DG Size | ∆ Size | Std Speed | DG Speed | |--------------|--------------|------------------|---------|----------|---------|--------|-----------|----------| | IQ2_XXS | 11.30 | 9.84 | -12.9% | 2.5G | 2.6G | +0.1G | 234s | 246s | | IQ2_XS | 11.72 | 11.63 | -0.8% | 2.7G | 2.8G | +0.1G | 242s | 246s | | IQ2_S | 14.31 | 9.02 | -36.9% | 2.7G | 2.9G | +0.2G | 238s | 244s | | IQ1_M | 27.46 | 15.41 | -43.9% | 2.2G | 2.5G | +0.3G | 206s | 212s | | IQ1_S | 53.07 | 32.00 | -39.7% | 2.1G | 2.4G | +0.3G | 184s | 209s |

• When to Use These Models:

  • Fitting models into GPU VRAM
  • Memory-constrained deployments
  • CPU and Edge Devices where 1 - 2 bit errors can be tolerated
  • Research into ultra-low-bit quantization

Choosing the Right Model Format

Included Files & Details

🔧 Technical Details

Model Architecture

• Devstral is an agentic LLM for software engineering tasks, built under a collaboration between Mistral AI and All Hands AI.
• It is finetuned from Mistral-Small-3.1, with a long context window of up to 128k tokens.

Benchmark Results

• SWE-Bench: Devstral achieves a score of 46.8% on SWE-Bench Verified, outperforming prior open-source SoTA by 6%. | Model | Scaffold | SWE-Bench Verified (%) | |------------------|--------------------|------------------------| | Devstral | OpenHands Scaffold | 46.8 | | GPT-4.1-mini | OpenAI Scaffold | 23.6 | | Claude 3.5 Haiku | Anthropic Scaffold | 40.6 | | SWE-smith-LM 32B | SWE-agent Scaffold | 40.2 |

📄 License

This project is licensed under the Apache-2.0 license.