Baichuan 7B

🚀 Baichuan-7B

Baichuan-7B is an open-source large-scale pre-trained model developed by Baichuan Intelligent Technology. It is based on the Transformer architecture and trained on approximately 1.2 trillion tokens. The model supports both Chinese and English, with a context window length of 4096, and achieves the best performance of its size on standard Chinese and English authoritative benchmarks.

🚀 Quick Start

If you wish to use Baichuan-7B (for inference, finetuning, etc.), we recommend using the accompanying code library Baichuan-7B.

✨ Features

• Among models of the same size, Baichuan-7B has achieved the current state-of-the-art (SOTA) level, as evidenced by the following MMLU metrics.
• Baichuan-7B is trained on proprietary bilingual Chinese-English corpora, optimized for Chinese, and achieves SOTA performance on C-Eval.
• Unlike LLaMA, which completely prohibits commercial use, Baichuan-7B employs a more lenient open-source license, allowing for commercial purposes.

💻 Usage Examples

Basic Usage

The following is a task of performing 1-shot inference using Baichuan-7B, where the author's name is given based on the work, with the correct output being "夜雨寄北->李商隐"

from transformers import AutoModelForCausalLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("baichuan-inc/Baichuan-7B", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("baichuan-inc/Baichuan-7B", device_map="auto", trust_remote_code=True)
inputs = tokenizer('登鹳雀楼->王之涣\n夜雨寄北->', return_tensors='pt')
inputs = inputs.to('cuda:0')
pred = model.generate(**inputs, max_new_tokens=64,repetition_penalty=1.1)
print(tokenizer.decode(pred.cpu()[0], skip_special_tokens=True))

Advanced Usage

The following is a task of performing 1-shot inference using Baichuan-7B, where the author's name is given based on the work, with the correct output being "One Hundred Years of Solitude->Gabriel Garcia Marquez"

from transformers import AutoModelForCausalLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("baichuan-inc/Baichuan-7B", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("baichuan-inc/Baichuan-7B", device_map="auto", trust_remote_code=True)
inputs = tokenizer('Hamlet->Shakespeare\nOne Hundred Years of Solitude->', return_tensors='pt')
inputs = inputs.to('cuda:0')
pred = model.generate(**inputs, max_new_tokens=64,repetition_penalty=1.1)
print(tokenizer.decode(pred.cpu()[0], skip_special_tokens=True))

📚 Documentation

Model Description

• Developed by: Baichuan Intelligent Technology
• Email: opensource@baichuan-inc.com
• Language(s) (NLP): Chinese/English
• License: Baichuan-7B License

Model Sources

The overall model is based on the standard Transformer structure, and we have adopted the same model design as LLaMA:

• Position Embedding: We use rotary-embedding, which is the position encoding scheme adopted by most models at this stage, and it has excellent extrapolation capabilities.
• Feedforward Layer: We use SwiGLU. The feedforward changes to (8/3) times the size of the hidden layer, that is, 11008.
• Layer Normalization: Pre-Normalization based on RMSNorm.

The specific parameters are as follows:

Uses

Downstream Use

We have also open-sourced the training code that accompanies this model, allowing for efficient finetuning for downstream tasks. For more details, please refer to Baichuan-7B.

Out-of-Scope Use

Production use without adequate assessment of risks and mitigation; any use cases which may be considered irresponsible or harmful.

Bias, Risks, and Limitations

Baichuan-7B can produce factually incorrect output, and should not be relied on to produce factually accurate information. Baichuan-7B was trained on various public datasets. While great efforts have been taken to clean the pretraining data, it is possible that this model could generate lewd, biased or otherwise offensive outputs.

Training Details

For specific training settings, please refer to Baichuan-7B.

Evaluation

Chinese Evaluation

C-Eval

CEval dataset is a comprehensive Chinese foundation model evaluation dataset, covering 52 disciplines and four difficulty levels. We used the dev set of this dataset as the source for few-shot, and conducted a 5-shot test on the test set.

Gaokao

Gaokao is a dataset that uses Chinese college entrance examination questions to evaluate the capabilities of large language models, aiming to assess the language and logical reasoning abilities of the models. We only retained the single-choice questions and conducted a unified 5-shot test on all models.

The following are the test results:

AGIEval

AGIEval aims to evaluate the general capabilities of models in cognitive and problem-solving related tasks. We only retained the four-choice single-choice questions, randomly divided them, and conducted a unified 5-shot test on all models.

^* The Aquila model is sourced from ZhiYuan official website, for reference only.

English Leaderboard

In addition to Chinese, we also tested the model's performance in English.

MMLU

MMLU is an English evaluation dataset that includes 57 multiple-choice tasks, covering elementary mathematics, American history, computer science, law, etc. The difficulty ranges from high school level to expert level, making it a mainstream LLM evaluation dataset.

We adopted the open-source evaluation scheme, and the final 5-shot results are as follows:

The superscript in the Model column indicates the source of the results.

0: reimplemented
1: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard
2: https://paperswithcode.com/sota/multi-task-language-understanding-on-mmlu

Our Group