Minicpm MoE 8x2B

🚀 MiniCPM-MoE-8x2B

MiniCPM-MoE-8x2B is a decoder-only generative language model based on the Transformer architecture, which adopts a Mixture-of-Experts (MoE) design to enhance performance.

🚀 Quick Start

The MiniCPM-MoE-8x2B is a decoder-only transformer-based generative language model. It adopts a Mixture-of-Experts (MoE) architecture, which has 8 experts per layer and activates 2 of 8 experts for each token.

You can access the OpenBMB Technical Blog Series for more information.

💻 Usage Examples

Basic Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
torch.manual_seed(0)

path = 'openbmb/MiniCPM-MoE-8x2B'
tokenizer = AutoTokenizer.from_pretrained(path)
model = AutoModelForCausalLM.from_pretrained(path, torch_dtype=torch.bfloat16, device_map='cuda', trust_remote_code=True)

responds, history = model.chat(tokenizer, "山东省最高的山是哪座山, 它比黄山高还是矮？差距多少？", temperature=0.8, top_p=0.8)
print(responds)

This is a model version after instruction tuning but without other rlhf methods. Chat template is automatically applied.

📄 Note

⚠️ Important Note

1. You can also perform inference with vLLM (>=0.4.1), which is compatible with this repo and has a much higher inference throughput.
2. The precision of model weights in this repo is bfloat16. Manual conversion is needed for other kinds of data types.
3. For more details, please refer to our GitHub repo.

📄 Statement

1. As a language model, MiniCPM-MoE-8x2B generates content by learning from a vast amount of text.
2. However, it does not possess the ability to comprehend or express personal opinions or value judgments.
3. Any content generated by MiniCPM-MoE-8x2B does not represent the viewpoints or positions of the model developers.
4. Therefore, when using content generated by MiniCPM-MoE-8x2B, users should take full responsibility for evaluating and verifying it on their own.