Colossal LLaMA 2 7b Base

🚀 Colossal-LLaMA-2-7B

🎉 The Colossal-AI team has released the open - source model Colossal-LLaMA-2-7B-base, which is based on LLaMA-2. This model can handle both Chinese and English, and has shown excellent performance in relevant evaluations.

🚀 Quick Start

To load the Colossal-LLaMA-2-7B-base model using Transformers, you can use the following code:

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("hpcai-tech/Colossal-LLaMA-2-7b-base", device_map="auto", trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained("hpcai-tech/Colossal-LLaMA-2-7b-base", trust_remote_code=True)
input = "明月松间照，\n\n->\n\n"
inputs = tokenizer(input, return_tensors='pt')
inputs = inputs.to('cuda:0')
pred = model.generate(**inputs,
                        max_new_tokens=512,
                        do_sample=True,
                        temperature=0.3,
                        top_k=50,
                        top_p=0.95,
                        num_return_sequences=1)
print(tokenizer.decode(pred.cpu()[0], skip_special_tokens=True)[len(input):])

✨ Features

• Open - source: The Colossal-LLaMA-2-7B-base model is open - source, which is beneficial for the research and development of the community.
• Multilingual Support: It supports both Chinese and English, with a context window of 4096 tokens.
• Cost - effective: With a pre - training cost of less than $1,000, it can achieve results similar to those of models that cost millions of dollars to pre - train from scratch.
• Good Performance: It has shown excellent performance in standard Chinese and English evaluation metrics such as C - Eval and MMLU.

💻 Usage Examples

Basic Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("hpcai-tech/Colossal-LLaMA-2-7b-base", device_map="auto", trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained("hpcai-tech/Colossal-LLaMA-2-7b-base", trust_remote_code=True)
input = "明月松间照，\n\n->\n\n"
inputs = tokenizer(input, return_tensors='pt')
inputs = inputs.to('cuda:0')
pred = model.generate(**inputs,
                        max_new_tokens=512,
                        do_sample=True,
                        temperature=0.3,
                        top_k=50,
                        top_p=0.95,
                        num_return_sequences=1)
print(tokenizer.decode(pred.cpu()[0], skip_special_tokens=True)[len(input):])

📚 Documentation

Performance Evaluation

We conducted comprehensive evaluations on 4 datasets and compared our Colossal-Llama-2-7b-base model with various models.

• We use 5 - shot for MMLU and calculate scores based on the logits of the first predicted token.
• We use 5 - shot for CMMLU and calculate scores based on the logits of the first predicted token.
• We use 5 - shot for AGIEval and only calculate scores for 4 - choice questions using a combination metric of exact match and the logits of the first predicted token. If any of the exact match or logits of the first predicted token is correct, the model will get the score.
• We use 0 - shot for GAOKAO - Bench and only calculate scores for 4 - choice questions based on the logits of the first predicted token.
• The generation config for all datasets is greedy search.
• We also provided CEval scores from its latest leaderboard or the official repository of the model.

The score in parentheses corresponds to the scores in the official repository of the model.

We use zero - shot for ChatGLM models.

To evaluate Qwen-7B on dataset MMLU, the prompt would be "xxx Answer:"(remove the space after ":") and we calculate the logits over " A", " B", " C" and " D" for Qwen-7B. Both the original and updated versions of Qwen-7B tend to be much more deterministic than other models. For example, the logits over " A" can be -inf and softmax would be exact 0.

For other models and other datasets, we calculate logits over "A", "B", "C" and "D".

❗️ More details of the evaluation methods and reproduction of the results, please refer to ColossalEval.

🔧 Technical Details

Data

Large language models such as LLaMA - 2 have been trained using a heterogeneous blend of high - quality datasets, yielding promising outcomes. Enhancing LLaMA - 2's performance for the Chinese corpus, while preserving its proficiency in English, critically hinges on two pivotal factors: the composition of the dataset, which encompasses both English and Chinese content, and the quality of each constituent dataset.

The following figure shows the data processing pipeline conducted for Colossal-LLaMA - 2.

❗️Important: We will open - source our data - processing toolkit soon, stay tuned!

Tokenizer

The original LLaMA - 2 vocabulary comprises fewer than a thousand Chinese characters, thus proving inadequate for encoding comprehensive Chinese texts effectively. Secondly, the utilization of byte tokens presents a challenge for transformer encoders to capture the semantic nuances of Chinese characters.

To address the above issues, we extend the LLaMA - 2 vocabulary from 32,000 to 69,104. To adapt the LLaMA - 2 model for use with the Colossal-LLaMA - 2 tokenizer, we initialize the new word embeddings by calculating the mean values from the original LLaMA - 2 embeddings and subsequently append these new rows to the end of the original embedding matrices.

Advantages of extending vocabulary size:

• Improve the compression rate of string sequence encoding.
• Enhance the integrity of information.
• Enable encoded sequences to contain more valuable information, thereby theoretically enhancing the ability for chapter - level encoding.

Advantages of large vocabulary size under low - resource settings:

• The presence of numerous unused tokens can be attributed to the limited training dataset, where an excessive number of tokens might not have been effectively learned.
• Excessive vocabulary expansion leads to an increase in embedding - related parameters, resulting in higher memory usage, which, in turn, affects the efficiency of the training process.

To balance both sides, we finally construct our vocabulary with size 69,104. The following table presents a comparison of various models at the 7B level.

Training Logs

Here are the training logs for our experiment:

Training Strategy

Multi - stage Training

In order to enhance the model's performance and harness the full potential of the original LLaMA - 2, we have developed a multi - stage training strategy. This strategy is designed to systematically unlock the model's capabilities over a series of stages.

Therefore, we have divided the training process into three stages:

• Large - scale pre - training stage (Conducted by LLaMA - 2): This initial stage is aimed at establishing the model's foundational capabilities from the ground up. It necessitates the use of a substantial dataset comprising no less than 1 trillion tokens.
• Chinese knowledge injection stage: In this stage, we introduce Chinese knowledge into the model. It requires access to a high - quality dataset rich in comprehensive knowledge relevant to the Chinese language.
• Knowledge replay stage: Knowledge is replayed through a question - answering (QA) mechanism, encompassing both the Chinese and English domains.

📄 License

This model is licensed under the LLaMA - 2 license and Apache 2.0 License without any additional commercial use restrictions.