Minicpm4 0.5B

🚀 MiniCPM4

MiniCPM4 is a highly efficient large language model series designed for end - side devices. It achieves remarkable efficiency improvements through innovations in model architecture, training data, training algorithms, and inference systems.

🚀 Quick Start

The MiniCPM4 series offers a range of models suitable for different end - side device requirements. You can choose from various models based on your needs, such as the flagship [MiniCPM4 - 8B](https://huggingface.co/openbmb/MiniCPM4 - 8B) with 8B parameters or the small - sized [MiniCPM4 - 0.5B](https://huggingface.co/openbmb/MiniCPM4 - 0.5B) with 0.5B parameters.

GitHub Repo | Technical Report

👋 Join us on Discord and WeChat

✨ Features

What's New

• [2025.06.06] MiniCPM4 series are released! This model achieves ultimate efficiency improvements while maintaining optimal performance at the same scale! It can achieve over 5x generation acceleration on typical end - side chips! You can find the technical report here.🔥🔥🔥

MiniCPM4 Series

The MiniCPM4 series consists of the following models:

• [MiniCPM4 - 8B](https://huggingface.co/openbmb/MiniCPM4 - 8B): The flagship of MiniCPM4, with 8B parameters, trained on 8T tokens.
• [MiniCPM4 - 0.5B](https://huggingface.co/openbmb/MiniCPM4 - 0.5B): The small version of MiniCPM4, with 0.5B parameters, trained on 1T tokens. (<-- you are here)
• [MiniCPM4 - 8B - Eagle - FRSpec](https://huggingface.co/openbmb/MiniCPM4 - 8B - Eagle - FRSpec): Eagle head for FRSpec, accelerating speculative inference for MiniCPM4 - 8B.
• [MiniCPM4 - 8B - Eagle - FRSpec - QAT - cpmcu](https://huggingface.co/openbmb/MiniCPM4 - 8B - Eagle - FRSpec - QAT - cpmcu): Eagle head trained with QAT for FRSpec, efficiently integrating speculation and quantization to achieve ultra - acceleration for MiniCPM4 - 8B.
• [MiniCPM4 - 8B - Eagle - vLLM](https://huggingface.co/openbmb/MiniCPM4 - 8B - Eagle - vLLM): Eagle head in vLLM format, accelerating speculative inference for MiniCPM4 - 8B.
• [MiniCPM4 - 8B - marlin - Eagle - vLLM](https://huggingface.co/openbmb/MiniCPM4 - 8B - marlin - Eagle - vLLM): Quantized Eagle head for vLLM format, accelerating speculative inference for MiniCPM4 - 8B.
• [BitCPM4 - 0.5B](https://huggingface.co/openbmb/BitCPM4 - 0.5B): Extreme ternary quantization applied to MiniCPM4 - 0.5B compresses model parameters into ternary values, achieving a 90% reduction in bit width.
• [BitCPM4 - 1B](https://huggingface.co/openbmb/BitCPM4 - 1B): Extreme ternary quantization applied to MiniCPM3 - 1B compresses model parameters into ternary values, achieving a 90% reduction in bit width.
• [MiniCPM4 - Survey](https://huggingface.co/openbmb/MiniCPM4 - Survey): Based on MiniCPM4 - 8B, accepts users' queries as input and autonomously generates trustworthy, long - form survey papers.
• [MiniCPM4 - MCP](https://huggingface.co/openbmb/MiniCPM4 - MCP): Based on MiniCPM4 - 8B, accepts users' queries and available MCP tools as input and autonomously calls relevant MCP tools to satisfy users' requirements.

Introduction

MiniCPM 4 is an extremely efficient edge - side large model that has undergone efficient optimization across four dimensions: model architecture, learning algorithms, training data, and inference systems, achieving ultimate efficiency improvements.

• 🏗️ Efficient Model Architecture:

  • InfLLM v2 -- Trainable Sparse Attention Mechanism: Adopts a trainable sparse attention mechanism architecture where each token only needs to compute relevance with less than 5% of tokens in 128K long text processing, significantly reducing computational overhead for long texts.

• 🧠 Efficient Learning Algorithms:

  • Model Wind Tunnel 2.0 -- Efficient Predictable Scaling: Introduces scaling prediction methods for performance of downstream tasks, enabling more precise model training configuration search.
  • BitCPM -- Ultimate Ternary Quantization: Compresses model parameter bit - width to 3 values, achieving 90% extreme model bit - width reduction.
  • Efficient Training Engineering Optimization: Adopts FP8 low - precision computing technology combined with Multi - token Prediction training strategy.

• 📚 High - Quality Training Data:

  • UltraClean -- High - quality Pre - training Data Filtering and Generation: Builds iterative data cleaning strategies based on efficient data verification, open - sourcing high - quality Chinese and English pre - training dataset [UltraFinweb](https://huggingface.co/datasets/openbmb/Ultra - FineWeb).
  • UltraChat v2 -- High - quality Supervised Fine - tuning Data Generation: Constructs large - scale high - quality supervised fine - tuning datasets covering multiple dimensions including knowledge - intensive data, reasoning - intensive data, instruction - following data, long text understanding data, and tool calling data.

• ⚡ Efficient Inference System:

  • CPM.cu -- Lightweight and Efficient CUDA Inference Framework: Integrates sparse attention, model quantization, and speculative sampling to achieve efficient prefilling and decoding.
  • ArkInfer -- Cross - platform Deployment System: Supports efficient deployment across multiple backend environments, providing flexible cross - platform adaptation capabilities.

📦 Installation

Inference with SGLang

For now, you need to install our forked version of SGLang.

git clone -b openbmb https://github.com/OpenBMB/sglang.git
cd sglang

pip install --upgrade pip
pip install -e "python[all]"

Inference with vLLM

For now, you need to install the latest version of vLLM.

pip install -U vllm \
    --pre \
    --extra-index-url https://wheels.vllm.ai/nightly

💻 Usage Examples

Basic Usage

Inference with Transformers

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
torch.manual_seed(0)

path = 'openbmb/MiniCPM4-0.5B'
device = "cuda"
tokenizer = AutoTokenizer.from_pretrained(path)
model = AutoModelForCausalLM.from_pretrained(path, torch_dtype=torch.bfloat16, device_map=device, trust_remote_code=True)

# User can directly use the chat interface
responds, history = model.chat(tokenizer, "Write an article about Artificial Intelligence.", temperature=0.7, top_p=0.7)
print(responds)

# User can also use the generate interface
# messages = [
#     {"role": "user", "content": "Write an article about Artificial Intelligence."},
# ]
# model_inputs = tokenizer.apply_chat_template(messages, return_tensors="pt", add_generation_prompt=True).to(device)

# model_outputs = model.generate(
#     model_inputs,
#     max_new_tokens=1024,
#     top_p=0.7,
#     temperature=0.7
# )
# output_token_ids = [
#     model_outputs[i][len(model_inputs[i]):] for i in range(len(model_inputs))
# ]

# responses = tokenizer.batch_decode(output_token_ids, skip_special_tokens=True)[0]
# print(responses)

Advanced Usage

Inference with SGLang

You can start the inference server by running the following command:

python -m sglang.launch_server --model openbmb/MiniCPM4-8B --trust-remote-code --port 30000 --chat-template chatml

Then you can use the chat interface by running the following command:

import openai

client = openai.Client(base_url=f"http://localhost:30000/v1", api_key="None")

response = client.chat.completions.create(
    model="openbmb/MiniCPM4-8B",
    messages=[
        {"role": "user", "content": "Write an article about Artificial Intelligence."},
    ],
    temperature=0.7,
    max_tokens=1024,
)

print(response.choices[0].message.content)

Inference with vLLM

from transformers import AutoTokenizer
from vllm import LLM, SamplingParams

model_name = "openbmb/MiniCPM4-8B"
prompt = [{"role": "user", "content": "Please recommend 5 tourist attractions in Beijing. "}]

tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
input_text = tokenizer.apply_chat_template(prompt, tokenize=False, add_generation_prompt=True)

llm = LLM(
    model=model_name,
    trust_remote_code=True,
    max_num_batched_tokens=32768, 
    dtype="bfloat16", 
    gpu_memory_utilization=0.8, 
)
sampling_params = SamplingParams(top_p=0.7, temperature=0.7, max_tokens=1024, repetition_penalty=1.02)

outputs = llm.generate(prompts=input_text, sampling_params=sampling_params)

print(outputs[0].outputs[0].text)

📚 Documentation

Evaluation Results

On two typical end - side chips, Jetson AGX Orin and RTX 4090, MiniCPM4 demonstrates significantly faster processing speed compared to similar - size models in long text processing tasks. As text length increases, MiniCPM4's efficiency advantage becomes more pronounced. On the Jetson AGX Orin platform, compared to Qwen3 - 8B, MiniCPM4 achieves approximately 7x decoding speed improvement.

Comprehensive Evaluation

MiniCPM4 launches end - side versions with 8B and 0.5B parameter scales, both achieving best - in - class performance in their respective categories.

Long Text Evaluation

MiniCPM4 is pre - trained on 32K long texts and achieves length extension through YaRN technology. In the 128K long text needle - in - a - haystack task, MiniCPM4 demonstrates outstanding performance.


Statement

• As a language model, MiniCPM generates content by learning from a vast amount of text.
• However, it does not possess the ability to comprehend or express personal opinions or value judgments.
• Any content generated by MiniCPM does not represent the viewpoints or positions of the model developers.
• Therefore, when using content generated by MiniCPM, users should take full responsibility for evaluating and verifying it on their own.

📄 License

• This repository and MiniCPM models are released under the Apache - 2.0 License.

📚 Citation

• Please cite our paper if you find our work valuable.

@article{minicpm4,
  title={{MiniCPM4}: Ultra-Efficient LLMs on End Devices},
  author={MiniCPM Team},
  year={2025}
}