Minicpm O 2 6

🚀 A GPT - 4o Level MLLM for Vision, Speech and Multimodal Live Streaming on Your Phone

MiniCPM - o 2.6 is an advanced model that integrates vision, speech, and multimodal live - streaming capabilities, offering high - performance and efficient solutions for various applications.

GitHub | Online Demo | Technical Blog

News

• [2025.03.01] 🚀🚀🚀 RLAIF - V, the alignment technique of MiniCPM - o, is accepted by CVPR 2025! The code, dataset, paper are open - sourced!
• [2025.01.24] 📢📢📢 MiniCPM - o 2.6 technical report is released! See Here.
• [2025.01.19] ⭐️⭐️⭐️ MiniCPM - o tops GitHub Trending and reaches top - 2 on Hugging Face Trending!

🚀 Quick Start

MiniCPM - o 2.6 can be easily used in various ways:

1. llama.cpp support for efficient CPU inference on local devices.
2. int4 and GGUF format quantized models in 16 sizes.
3. vLLM support for high - throughput and memory - efficient inference.
4. Fine - tuning on new domains and tasks with LLaMA - Factory.
5. Quick local WebUI demo setup with Gradio.
6. Online web demo on server.

✨ Features

MiniCPM - o 2.6 Features

MiniCPM - o 2.6 is the latest and most capable model in the MiniCPM - o series. It is built end - to - end based on SigLip - 400M, Whisper - medium - 300M, ChatTTS - 200M, and Qwen2.5 - 7B with a total of 8B parameters. It shows significant performance improvement over MiniCPM - V 2.6 and introduces new features for real - time speech conversation and multimodal live streaming.

• 🔥 Leading Visual Capability
  • MiniCPM - o 2.6 achieves an average score of 70.2 on OpenCompass, a comprehensive evaluation over 8 popular benchmarks. With only 8B parameters, it surpasses widely used proprietary models like GPT - 4o - 202405, Gemini 1.5 Pro, and Claude 3.5 Sonnet for single image understanding. It also outperforms GPT - 4V and Claude 3.5 Sonnet in multi - image and video understanding and shows promising in - context learning capability.
• 🎙 State - of - the - art Speech Capability
  • MiniCPM - o 2.6 supports bilingual real - time speech conversation with configurable voices in English and Chinese. It outperforms GPT - 4o - realtime on audio understanding tasks such as ASR and STT translation and shows state - of - the - art performance on speech conversation in both semantic and acoustic evaluations in the open - source community. It also allows for fun features such as emotion/speed/style control, end - to - end voice cloning, role play, etc.
• 🎬 Strong Multimodal Live Streaming Capability
  • As a new feature, MiniCPM - o 2.6 can accept continuous video and audio streams independent of user queries and support real - time speech interaction. It outperforms GPT - 4o - 202408 and Claude 3.5 Sonnet and shows state - of - the - art performance in the open - source community on StreamingBench, a comprehensive benchmark for real - time video understanding, omni - source (video & audio) understanding, and multimodal contextual understanding.
• 💪 Strong OCR Capability and Others
  • Advancing popular visual capabilities from the MiniCPM - V series, MiniCPM - o 2.6 can process images with any aspect ratio and up to 1.8 million pixels (e.g., 1344x1344). It achieves state - of - the - art performance on OCRBench for models under 25B, surpassing proprietary models such as GPT - 4o - 202405.
  • Based on the latest RLAIF - V and VisCPM techniques, it features trustworthy behaviors, outperforming GPT - 4o and Claude 3.5 Sonnet on MMHal - Bench, and supports multilingual capabilities on more than 30 languages.
• 🚀 Superior Efficiency
  • In addition to its friendly size, MiniCPM - o 2.6 also shows state - of - the - art token density (i.e., number of pixels encoded into each visual token). It produces only 640 tokens when processing a 1.8M pixel image, which is 75% fewer than most models. This directly improves the inference speed, first - token latency, memory usage, and power consumption. As a result, MiniCPM - o 2.6 can efficiently support multimodal live streaming on end - side devices such as iPad.
• 💫 Easy Usage
  • MiniCPM - o 2.6 can be easily used in various ways as mentioned in the Quick Start section.

Model Architecture

• End - to - end Omni - modal Architecture
  • Different modality encoder/decoders are connected and trained in an end - to - end fashion to fully exploit rich multimodal knowledge.
• Omni - modal Live Streaming Mechanism
  • (1) We change the offline modality encoder/decoders into online ones for streaming inputs/outputs.
  • (2) We devise a time - division multiplexing (TDM) mechanism for omni - modality streaming processing in the LLM backbone. It divides parallel omni - modality streams into sequential info within small periodic time slices.
• Configurable Speech Modeling Design
  • We devise a multimodal system prompt, including a traditional text system prompt and a new audio system prompt to determine the assistant voice. This enables flexible voice configurations in inference time and also facilitates end - to - end voice cloning and description - based voice creation.

📚 Documentation

Evaluation

Visual understanding results

Image Understanding:

⁺ Token Density: number of pixels encoded into each visual token at maximum resolution, i.e., # pixels at maximum resolution / # visual tokens.

Note: For proprietary models, we calculate token density based on the image encoding charging strategy defined in the official API documentation, which provides an upper - bound estimation.

Multi - image and Video Understanding:

Audio understanding and speech conversation results.

Audio Understanding: