MiniCPM4-8B-GGUF Open-Source Large Language Model - Suitable for Edge Devices, Generation Speed Increased by Over 5 Times

Minicpm4 8B GGUF

Developed by openbmb

MiniCPM4 is an efficient large language model designed specifically for edge devices. While maintaining optimal performance at the same scale, it achieves extreme efficiency improvements, enabling over 5x generation acceleration on typical edge chips.

Large Language Model

Transformers

Supports Multiple LanguagesOpen Source License:Apache-2.0 #Efficient inference on the edge #Ternary quantization compression #Sparse attention for long texts

Downloads 324

Release Time : 6/13/2025

Model Overview

MiniCPM4 is an efficient large language model optimized for edge devices. Through innovations in four dimensions: model architecture, training data, training algorithms, and inference systems, it achieves a balance between high performance and high efficiency.

Model Features

Efficient model architecture

Adopts a trainable sparse attention mechanism architecture. In the processing of 128K long texts, each token only needs to calculate the correlation with less than 5% of the tokens, significantly reducing the computational overhead of long texts.

Efficient learning algorithm

Introduces a scaling prediction method for downstream task performance to achieve a more accurate search for model training configurations; uses FP8 low - precision computing technology combined with a multi - token prediction training strategy.

High - quality training data

Builds an iterative data cleaning strategy based on efficient data validation, using high - quality Chinese and English pre - training datasets UltraFinweb and large - scale supervised fine - tuning dataset UltraChat v2.

Efficient inference system

Integrates sparse attention, model quantization, and speculative sampling to achieve efficient pre - filling and decoding; supports efficient deployment in multiple backend environments.

Extreme quantization technology

Compresses the model parameter bit - width to 3 values through BitCPM technology, achieving a 90% extreme reduction in model bit - width.

Model Capabilities

Text generation

Long - text understanding

Tool invocation

Investigation paper generation

Speculative inference acceleration

Use Cases

Content generation

Investigation paper generation

Autonomously generate credible long - form investigation papers based on user queries

Efficiency optimization

Speculative inference acceleration

Achieve over 5x generation acceleration through Eagle head and FRSpec technology

Achieve over 5x acceleration on typical edge chips

Edge computing

Edge deployment

Efficient inference optimized for edge devices

Maintain high performance on resource - constrained devices

🚀 MiniCPM4

MiniCPM4 is a highly efficient large language model series designed for end - side devices, achieving remarkable efficiency improvements through innovations in multiple dimensions.

🚀 Quick Start

MiniCPM4 series offer a range of models suitable for various end - side device scenarios. You can quickly start using them by referring to the model list and usage examples below.

✨ Features

What's New

[2025.06.06] The MiniCPM4 series are released! This model achieves ultimate efficiency improvements while maintaining optimal performance at the same scale! It can achieve over 5x generation acceleration on typical end - side chips! You can find the technical report here.

MiniCPM4 Series

The MiniCPM4 series are highly efficient large language models (LLMs) designed explicitly for end - side devices, which achieves this efficiency through systematic innovation in four key dimensions: model architecture, training data, training algorithms, and inference systems.

[MiniCPM4 - 8B](https://huggingface.co/openbmb/MiniCPM4 - 8B): The flagship of MiniCPM4, with 8B parameters, trained on 8T tokens.
[MiniCPM4 - 0.5B](https://huggingface.co/openbmb/MiniCPM4 - 0.5B): The small version of MiniCPM4, with 0.5B parameters, trained on 1T tokens.
[MiniCPM4 - 8B - Eagle - FRSpec](https://huggingface.co/openbmb/MiniCPM4 - 8B - Eagle - FRSpec): Eagle head for FRSpec, accelerating speculative inference for MiniCPM4 - 8B.
[MiniCPM4 - 8B - Eagle - FRSpec - QAT - cpmcu](https://huggingface.co/openbmb/MiniCPM4 - 8B - Eagle - FRSpec - QAT - cpmcu): Eagle head trained with QAT for FRSpec, efficiently integrating speculation and quantization to achieve ultra - acceleration for MiniCPM4 - 8B.
[MiniCPM4 - 8B - Eagle - vLLM](https://huggingface.co/openbmb/MiniCPM4 - 8B - Eagle - vLLM): Eagle head in vLLM format, accelerating speculative inference for MiniCPM4 - 8B.
[MiniCPM4 - 8B - marlin - Eagle - vLLM](https://huggingface.co/openbmb/MiniCPM4 - 8B - marlin - Eagle - vLLM): Quantized Eagle head for vLLM format, accelerating speculative inference for MiniCPM4 - 8B.
[BitCPM4 - 0.5B](https://huggingface.co/openbmb/BitCPM4 - 0.5B): Extreme ternary quantization applied to MiniCPM4 - 0.5B compresses model parameters into ternary values, achieving a 90% reduction in bit width.
[BitCPM4 - 1B](https://huggingface.co/openbmb/BitCPM4 - 1B): Extreme ternary quantization applied to MiniCPM3 - 1B compresses model parameters into ternary values, achieving a 90% reduction in bit width.
[MiniCPM4 - Survey](https://huggingface.co/openbmb/MiniCPM4 - Survey): Based on MiniCPM4 - 8B, accepts users' queries as input and autonomously generates trustworthy, long - form survey papers.
[MiniCPM4 - MCP](https://huggingface.co/openbmb/MiniCPM4 - MCP): Based on MiniCPM4 - 8B, accepts users' queries and available MCP tools as input and autonomously calls relevant MCP tools to satisfy users' requirements.
[MiniCPM4 - 8B - GGUF](https://huggingface.co/openbmb/MiniCPM4 - 8B - GGUF): GGUF version of MiniCPM4 - 8B. (<-- you are here)

Introduction

MiniCPM 4 is an extremely efficient edge - side large model that has undergone efficient optimization across four dimensions: model architecture, learning algorithms, training data, and inference systems, achieving ultimate efficiency improvements.

Efficient Model Architecture:
- InfLLM v2 -- Trainable Sparse Attention Mechanism: Adopts a trainable sparse attention mechanism architecture where each token only needs to compute relevance with less than 5% of tokens in 128K long text processing, significantly reducing computational overhead for long texts.
Efficient Learning Algorithms:
- Model Wind Tunnel 2.0 -- Efficient Predictable Scaling: Introduces scaling prediction methods for performance of downstream tasks, enabling more precise model training configuration search.
- BitCPM -- Ultimate Ternary Quantization: Compresses model parameter bit - width to 3 values, achieving 90% extreme model bit - width reduction.
- Efficient Training Engineering Optimization: Adopts FP8 low - precision computing technology combined with Multi - token Prediction training strategy.
High - Quality Training Data:
- UltraClean -- High - quality Pre - training Data Filtering and Generation: Builds iterative data cleaning strategies based on efficient data verification, open - sourcing high - quality Chinese and English pre - training dataset [UltraFinweb](https://huggingface.co/datasets/openbmb/Ultra - FineWeb).
- UltraChat v2 -- High - quality Supervised Fine - tuning Data Generation: Constructs large - scale high - quality supervised fine - tuning datasets covering multiple dimensions including knowledge - intensive data, reasoning - intensive data, instruction - following data, long text understanding data, and tool calling data.
Efficient Inference System:
- CPM.cu -- Lightweight and Efficient CUDA Inference Framework: Integrates sparse attention, model quantization, and speculative sampling to achieve efficient prefilling and decoding.
- ArkInfer -- Cross - platform Deployment System: Supports efficient deployment across multiple backend environments, providing flexible cross - platform adaptation capabilities.

💻 Usage Examples

Basic Usage

Inference with llama.cpp

./llama-cli -c 1024 -m MiniCPM4-8B-Q4_K_M.gguf -n 1024 --top-p 0.7 --temp 0.7 --prompt "<|im_start|>user\nËØ∑ÂÜô‰∏ÄÁØáÂÖ≥‰∫é‰∫∫Â∑•Êô∫ËÉΩÁöÑÊñáÁ´†ÔºåËØ¶ÁªÜ‰ªãÁªç‰∫∫Â∑•Êô∫ËÉΩÁöÑÊú™Êù•ÂèëÂ±ïÂíåÈöêÊÇ£„ÄÇ<|im_end|>\n<|im_start|>assistant\n"

📚 Documentation

Statement

As a language model, MiniCPM generates content by learning from a vast amount of text.
However, it does not possess the ability to comprehend or express personal opinions or value judgments.
Any content generated by MiniCPM does not represent the viewpoints or positions of the model developers.
Therefore, when using content generated by MiniCPM, users should take full responsibility for evaluating and verifying it on their own.

📄 License

This repository and MiniCPM models are released under the Apache - 2.0 License.

📚 Documentation

Citation

Please cite our paper if you find our work valuable.

@article{minicpm4,
  title={{MiniCPM4}: Ultra-Efficient LLMs on End Devices},
  author={MiniCPM Team},
  year={2025}
}

GitHub Repo | Technical Report

Join us on Discord and WeChat

Featured Recommended AI Models

Empowering the Future, Your AI Solution Knowledge Base

English 简体中文繁體中文にほんご