Minicpm Embedding

🚀 MiniCPM-Embedding

MiniCPM-Embedding is a bilingual Chinese-English text embedding model jointly developed by ModelBest Inc., the Natural Language Processing Laboratory of Tsinghua University (THUNLP), and the Information Retrieval Group of Northeastern University (NEUIR). It offers the following features:

• Exceptional retrieval capabilities for both Chinese and English texts.
• Outstanding cross - lingual retrieval capabilities between Chinese and English.

MiniCPM-Embedding is trained based on MiniCPM-2B-sft-bf16. Structurally, it adopts bidirectional attention and Weighted Mean Pooling [1]. It uses a multi - stage training approach, with approximately 6 million training data, including open - source data, machine - generated data, and closed - source data.

We invite you to explore the RAG toolkit series:

• Retrieval Model: MiniCPM-Embedding
• Re - ranking Model: MiniCPM-Reranker
• LoRA Plugin for RAG scenarios: MiniCPM3-RAG-LoRA

[1] Muennighoff, N. (2022). Sgpt: Gpt sentence embeddings for semantic search. arXiv preprint arXiv:2202.08904.

✨ Features

• Bilingual Retrieval: Exceptional retrieval capabilities for both Chinese and English texts.
• Cross - lingual Retrieval: Outstanding cross - lingual retrieval capabilities between Chinese and English.

📦 Installation

Requirements

transformers==4.37.2

💻 Usage Examples

Basic Usage

Huggingface Transformers

from transformers import AutoModel, AutoTokenizer
import torch
import torch.nn.functional as F

model_name = "openbmb/MiniCPM-Embedding"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModel.from_pretrained(model_name, trust_remote_code=True, torch_dtype=torch.float16).to("cuda")
# You can also use the following line to enable the Flash Attention 2 implementation
# model = AutoModel.from_pretrained(model_name, trust_remote_code=True, attn_implementation="flash_attention_2", torch_dtype=torch.float16).to("cuda")
model.eval()

# As we scale hidden states in `model.forward`, mean pooling here actually works as weighted mean pooling
def mean_pooling(hidden, attention_mask):
    s = torch.sum(hidden * attention_mask.unsqueeze(-1).float(), dim=1)
    d = attention_mask.sum(dim=1, keepdim=True).float()
    reps = s / d
    return reps

@torch.no_grad()
def encode(input_texts):
    batch_dict = tokenizer(input_texts, max_length=512, padding=True, truncation=True, return_tensors='pt', return_attention_mask=True).to("cuda")
    
    outputs = model(**batch_dict)
    attention_mask = batch_dict["attention_mask"]
    hidden = outputs.last_hidden_state

    reps = mean_pooling(hidden, attention_mask)   
    embeddings = F.normalize(reps, p=2, dim=1).detach().cpu().numpy()
    return embeddings

queries = ["中国的首都是哪里？"]
passages = ["beijing", "shanghai"]


INSTRUCTION = "Query: "
queries = [INSTRUCTION + query for query in queries]

embeddings_query = encode(queries)
embeddings_doc = encode(passages)

scores = (embeddings_query @ embeddings_doc.T)
print(scores.tolist())  # [[0.3535913825035095, 0.18596848845481873]]

Sentence Transformers

import torch
from sentence_transformers import SentenceTransformer

model_name = "openbmb/MiniCPM-Embedding"
model = SentenceTransformer(model_name, trust_remote_code=True, model_kwargs={ "torch_dtype": torch.float16})
# You can also use the following line to enable the Flash Attention 2 implementation
# model = SentenceTransformer(model_name, trust_remote_code=True, attn_implementation="flash_attention_2", model_kwargs={ "torch_dtype": torch.float16})

queries = ["中国的首都是哪里？"]
passages = ["beijing", "shanghai"]

INSTRUCTION = "Query: "

embeddings_query = model.encode(queries, prompt=INSTRUCTION)
embeddings_doc = model.encode(passages)

scores = (embeddings_query @ embeddings_doc.T)
print(scores.tolist())  # [[0.35365450382232666, 0.18592746555805206]]

Input Format

This model supports query - side instructions in the following format:

Instruction: {{ instruction }} Query: {{ query }}

For example:

Instruction: 为这个医学问题检索相关回答。Query: 咽喉癌的成因是什么？

Instruction: Given a claim about climate change, retrieve documents that support or refute the claim. Query: However the warming trend is slower than most climate models have forecast.

It also works in instruction - free mode in the following format:

Query: {{ query }}

When running evaluation on BEIR and C - MTEB/Retrieval, we use instructions in instructions.json. For other evaluations, we do not use instructions. On the document side, we directly use the bare document as the input.

📚 Documentation

Model Information

Evaluation Results

CN/EN Retrieval Results

CN - EN Cross - lingual Retrieval Results