MiniCPM-V 2.6 Open-source Multimodal Vision-Language Model - Supports Image-Text to Text Conversion and Multilingual Processing

Minicpm V 2 6 Int4

Developed by openbmb

MiniCPM-V 2.6 is a multimodal vision-language model supporting image-to-text conversion with multilingual processing capabilities.

Image-to-Text

Transformers

Other#Multimodal Live Streaming #Real-time Voice Chat #Multilingual Support

Downloads 122.58k

Release Time : 8/4/2024

Model Overview

MiniCPM-V 2.6 is a multimodal model based on the MiniCPM-V architecture, focusing on vision-language tasks. It can process various inputs such as images, text, and videos, and generate corresponding text outputs.

Model Features

Multimodal Support

Supports various input modalities such as images, text, and videos, capable of handling complex multimodal tasks.

Multilingual Processing

Supports multiple languages with cross-lingual processing capabilities.

High Performance

Significant performance improvement over previous models, supporting real-time processing.

Model Capabilities

Image-to-Text Conversion

Multilingual Text Generation

Video Content Analysis

Optical Character Recognition

Multi-Image Processing

Use Cases

Content Generation

Image Captioning

Generates detailed textual descriptions based on input images.

Produces accurate and detailed image captions.

Video Summarization

Analyzes video content and generates concise textual summaries.

Generates text summaries of video content for quick understanding.

Document Processing

Optical Character Recognition

Extracts text information from images or videos.

High-precision text recognition and extraction.

🚀 MiniCPM-V 2.6 int4

This is an int4 quantized version of MiniCPM-V 2.6, which can reduce GPU memory usage.

🚀 Quick Start

This project is an int4 quantized version of MiniCPM-V 2.6. Running with the int4 version can significantly reduce GPU memory usage, requiring only about 7GB.

✨ Features

Pipeline Tag: image-text-to-text
Datasets: openbmb/RLAIF-V-Dataset
Library Name: transformers
Language: Multilingual
Tags: minicpm-v, vision, ocr, multi-image, video, custom_code
Base Model: openbmb/MiniCPM-V-2_6

Property	Details
Pipeline Tag	image-text-to-text
Datasets	openbmb/RLAIF-V-Dataset
Library Name	transformers
Language	Multilingual
Tags	minicpm-v, vision, ocr, multi-image, video, custom_code
Base Model	openbmb/MiniCPM-V-2_6

📦 Installation

Inference using Huggingface transformers on NVIDIA GPUs. Requirements tested on python 3.10:

Pillow==10.1.0
torch==2.1.2
torchvision==0.16.2
transformers==4.40.0
sentencepiece==0.1.99
accelerate==0.30.1
bitsandbytes==0.43.1

💻 Usage Examples

Basic Usage

# test.py
import torch
from PIL import Image
from transformers import AutoModel, AutoTokenizer

model = AutoModel.from_pretrained('openbmb/MiniCPM-V-2_6-int4', trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained('openbmb/MiniCPM-V-2_6-int4', trust_remote_code=True)
model.eval()

image = Image.open('xx.jpg').convert('RGB')
question = 'What is in the image?'
msgs = [{'role': 'user', 'content': [image, question]}]

res = model.chat(
    image=None,
    msgs=msgs,
    tokenizer=tokenizer
)
print(res)

Advanced Usage

## if you want to use streaming, please make sure sampling=True and stream=True
## the model.chat will return a generator
res = model.chat(
    image=None,
    msgs=msgs,
    tokenizer=tokenizer,
    sampling=True,
    temperature=0.7,
    stream=True
)

generated_text = ""
for new_text in res:
    generated_text += new_text
    print(new_text, flush=True, end='')

📚 Documentation

News

[2025.01.14] 🔥🔥 We open source MiniCPM-o 2.6, with significant performance improvement over MiniCPM-V 2.6, and support real-time speech-to-speech conversation and multimodal live streaming. Try it now.

Featured Recommended AI Models

Empowering the Future, Your AI Solution Knowledge Base

English 简体中文繁體中文にほんご