Devstral-Small-2505-GGUF Open-source Model - Multiple Precision Options to Adapt to Different Hardware Requirements

Devstral Small 2505 GGUF

Developed by Antigma

Quantized version of Devstral-Small-2505, offering multiple precision options to adapt to different hardware requirements

Large Language Model Supports Multiple LanguagesOpen Source License:Apache-2.0 #Multi-precision Quantization #Lightweight Deployment #Edge Computing Optimization

Downloads 170

Release Time : 5/22/2025

Model Overview

This model is the GGUF quantized version of Devstral-Small-2505, suitable for local inference scenarios, providing multiple quantization precision options from 2-bit to 8-bit to balance model quality and computational resource consumption

Model Features

Multi-level Quantization Options

Offers 6 quantization levels from Q2_K to Q8_0 to meet precision and performance needs in various scenarios

Strong Hardware Adaptability

Quantized models significantly reduce memory usage, enabling the model to run on consumer-grade hardware

Efficient Inference

Optimizes inference speed through quantization techniques while maintaining acceptable model quality

Model Capabilities

Text Generation

Local Inference

Use Cases

Local Applications

Personal Assistant

Deploy personalized AI assistants on local devices

Low-latency response, privacy protection

Content Creation

Supports creative writing and content generation in offline environments

Balances generation quality with resource consumption

Research & Development

Model Quantization Research

Study the impact of different quantization levels on model performance

Provides comparisons across multiple quantization levels

🚀 Devstral-Small-2505 GGUF Quantized Model

This project provides quantized versions of the Devstral-Small-2505 model, enabling efficient deployment and inference. It supports multiple languages and is quantized using llama.cpp.

🚀 Quick Start

Model Information

Property	Details
Base Model	unsloth/Devstral-Small-2505
Supported Languages	en, fr, de, es, pt, it, ja, ko, ru, zh, ar, fa, id, ms, ne, pl, ro, sr, sv, tr, uk, vi, hi, bn
Library Name	vllm
License	apache-2.0
Pipeline Tag	text2text-generation
Tags	llama-cpp, gguf-my-repo
Inference	false

About the Producer

Produced by Antigma Labs, Antigma Quantize Space

Follow Antigma Labs on X https://x.com/antigma_labs

Antigma's GitHub Homepage https://github.com/AntigmaLabs

Privacy Policy

⚠️ Important Note

If you want to learn more about how we process your personal data, please read our Privacy Policy.

✨ Features

llama.cpp Quantization

Using llama.cpp release b4944 for quantization. The original model can be found at https://huggingface.co/unsloth/Devstral-Small-2505. You can run the quantized models directly with llama.cpp or any other llama.cpp based project.

Prompt Format

<｜begin▁of▁sentence｜>{system_prompt}<｜User｜>{prompt}<｜Assistant｜><｜end▁of▁sentence｜><｜Assistant｜>

📦 Installation

Download a File

You can download a specific file from the following table:

Filename	Quant type	File Size	Split
devstral-small-2505-q2_k.gguf	Q2_K	8.28 GB	False
devstral-small-2505-q3_k_l.gguf	Q3_K_L	11.55 GB	False
devstral-small-2505-q6_k.gguf	Q6_K	18.02 GB	False
devstral-small-2505-q4_k_m.gguf	Q4_K_M	13.35 GB	False
devstral-small-2505-q5_k_m.gguf	Q5_K_M	15.61 GB	False
devstral-small-2505-q8_0.gguf	Q8_0	23.33 GB	False

Downloading using huggingface-cli

Click to view download instructions

First, make sure you have hugginface-cli installed:

pip install -U "huggingface_hub[cli]"

Then, you can target the specific file you want:

huggingface-cli download https://huggingface.co/Antigma/Devstral-Small-2505-GGUF --include "devstral-small-2505-q2_k.gguf" --local-dir ./

If the model is bigger than 50GB, it will have been split into multiple files. In order to download them all to a local folder, run:

huggingface-cli download https://huggingface.co/Antigma/Devstral-Small-2505-GGUF --include "devstral-small-2505-q2_k.gguf/*" --local-dir ./

You can either specify a new local-dir (deepseek-ai_DeepSeek-V3-0324-Q8_0) or download them all in place (./)

📄 License

This project is licensed under the apache-2.0 license.

Featured Recommended AI Models

Empowering the Future, Your AI Solution Knowledge Base

English 简体中文繁體中文にほんご