Lingowhale 8B

🚀 LingoWhale-8B

LingoWhale-8B is the first open-source bilingual large language model in Chinese and English in the LingoWhale series launched by DeepLangAI. It has powerful basic capabilities and achieves leading results on multiple public evaluation benchmarks.

🚀 Quick Start

This README provides detailed information about the LingoWhale-8B model, including model introduction, evaluation results, generation examples, deployment and inference, fine-tuning methods, and open-source licenses. You can quickly understand and use the LingoWhale-8B model based on the content in this document.

✨ Features

• Powerful Foundation: Pre-trained on trillions of high-quality Chinese and English tokens, achieving leading results on multiple public evaluation benchmarks.
• Long Context Handling: Trained with an 8K context length, capable of handling longer context understanding and generation tasks.
• Academic and Commercial Use: Completely open for academic research. After applying by email and obtaining official commercial licenses, users can use it for free commercially.
• Developer-Friendly: Provides Huggingface inference interfaces and parameter-efficient fine-tuning examples like LoRA for quick development.

📦 Installation

Install Dependencies

pip install -r requirements.txt

• If you need to use flash attention, install flash-attn.
• If you need to use parameter-efficient fine-tuning methods like LoRA, install peft additionally.

💻 Usage Examples

Basic Usage

>>> from transformers import AutoModelForCausalLM, AutoTokenizer
>>> tokenizer = AutoTokenizer.from_pretrained("deeplang-ai/LingoWhale-8B", trust_remote_code=True)
>>> model = AutoModelForCausalLM.from_pretrained("deeplang-ai/LingoWhale-8B", device_map="auto", trust_remote_code=True)
>>> inputs = tokenizer("陋室铭\n唐 刘禹锡\n", return_tensors="pt")
>>> inputs = inputs.to("cuda:0")
>>> pred = model.generate(**inputs, max_new_tokens=100, repetition_penalty=1.1)
>>> print(tokenizer.decode(pred.cpu()[0], skip_special_tokens=True))

Advanced Usage

Here are some generation examples of the LingoWhale-8B model:

import re

def is_valid_phone(phone):
    pattern = r'^1[3-9]\d{9}$'
    return bool(re.match(pattern, phone))

print(is_valid_phone('13800138000'))  # True
print(is_valid_phone('12345678901'))  # False

📚 Documentation

Model Introduction

LingoWhale-8B is the first open-source bilingual large language model in Chinese and English in the LingoWhale series launched by DeepLangAI. The model is pre-trained on trillions of high-quality Chinese and English tokens, endowing it with powerful basic capabilities and achieving leading results on multiple public evaluation benchmarks. During the pre-training phase, the model is trained with an 8K context length, enabling it to handle longer context understanding and generation tasks.

The LingoWhale-8B model is completely open for academic research. After users apply by email and obtain official commercial licenses, they can use it for free commercially. While open-sourcing the model weights, we also provide Huggingface inference interfaces that conform to users' habits and parameter-efficient fine-tuning examples like LoRA, facilitating developers to quickly use the LingoWhale-8B model.

Due to the influence of the model's parameter quantity, inherent problems of large models such as hallucination and relatively weak mathematical calculation ability still exist in the LingoWhale-8B model. Please understand these issues before use and evaluate the possible risks. Future versions of the LingoWhale model will focus on optimizing such problems.

Evaluation Results

We conducted tests on the following public evaluation benchmarks:

• C-Eval is a Chinese basic model evaluation benchmark containing 13,948 multiple-choice questions covering 52 different disciplines and four difficulty levels. It aims to evaluate the capabilities of Chinese language models. We used the dev set of this dataset as the source for few-shot learning and conducted 5-shot tests on the test set.
• MMLU is an English basic model evaluation benchmark covering multiple fields such as basic mathematics, American history, computer science, and law, with a total of 57 tasks. It is used to evaluate the performance of language models on tasks in different fields. We conducted 5-shot tests on the model.
• CMMLU is a Chinese evaluation benchmark covering 67 topics from basic disciplines to advanced professional levels. It is used to evaluate the knowledge and reasoning abilities of Chinese language models. We used the dev set of this dataset as the source for few-shot learning and conducted 5-shot tests on the test set.
• Gaokao is an evaluation benchmark using Chinese college entrance examination questions as the dataset. It aims to evaluate the language understanding and logical reasoning abilities of Chinese language models. We only retained the four-choice multiple-choice questions and conducted 5-shot tests on the model after random division.
• HumanEval is an English evaluation benchmark containing hundreds of programming questions. It is used to evaluate the program understanding and generation abilities of language models. We used the zero-shot method to calculate Pass@1 and tested the model.
• GSM8K is a dataset composed of high-quality and linguistically diverse primary school math word problems. It requires selecting the most reasonable solution based on a given scenario and is used to evaluate the mathematical application abilities of language models. We conducted 8-shot tests on the model.
• BBH is an evaluation benchmark formed by selecting poorly performing tasks from 204 Big-Bench evaluation benchmark tasks. It is used to evaluate the performance of large language models on challenging tasks. We conducted 3-shot tests on the model.
• AGIEval is a benchmark test examining the human-like abilities of basic models, specifically used to evaluate the abilities of basic models in tasks related to human cognition and problem-solving. We only retained the four-choice multiple-choice questions and conducted 5-shot tests on the model after random division.

These evaluation benchmarks provide standardized tests and metrics for evaluating the performance and capabilities of language models on different tasks and in different fields. The evaluation methods and results are shown in the following table:

* indicates that the model results are from the official source, and all results are accurate to one decimal place.

Fine-tuning Methods

The sample code for model fine-tuning is based on transformers.Trainer. Most of the parameters and usage methods can refer to the tutorials and introductions of Trainer on Huggingface.

This section aims to demonstrate the fine-tuning process and does not guarantee the performance of the model after fine-tuning under this fine-tuning configuration.

Single-machine Training

The following is an example of single-machine fine-tuning. The data used is 10,000 instruction fine-tuning data randomly selected from the COIG dataset. You can replace it with your own data.

hostfile=""
deepspeed --hostfile=$hostfile finetune/finetune.py  \
    --report_to "none" \
    --data_path "finetune/data/coig_10k.json" \
    --model_name_or_path deeplang-ai/LingoWhale-8B \
    --output_dir "output" \
    --model_max_length 2048 \
    --num_train_epochs 4 \
    --per_device_train_batch_size 4 \
    --gradient_accumulation_steps 1 \
    --save_strategy epoch \
    --learning_rate 2e-5 \
    --lr_scheduler_type constant \
    --adam_beta1 0.9 \
    --adam_beta2 0.98 \
    --adam_epsilon 1e-8 \
    --max_grad_norm 1.0 \
    --weight_decay 1e-4 \
    --warmup_ratio 0.0 \
    --logging_steps 1 \
    --gradient_checkpointing True \
    --deepspeed finetune/ds_config.json \
    --bf16 True \
    --tf32 True

If you want to replace it with your own data, you can use a json file in the following format.

[
    {
        "id": 0,
        "conversations": [
            {
                "from": "human",
                "value": "What is the 'modal age'?"
            },
            {
                "from": "model",
                "value": "The modal age refers to the age calculated by assuming the initial isotopic composition of geological samples using the law of radioactive decay."
            },
            ...
        ]
    },
    ...
]

Multi-machine Training

For multi-machine training, you need to edit a hostfile file in the following format. Each line represents a machine, where ip_address-X is the IP address of each machine, and the slots field indicates the number of available GPUs on the machine. The content format is as follows:

ip_address-1 slots=8
ip_address-2 slots=8
ip_address-3 slots=8
ip_address-4 slots=8
...

At the same time, specify the hostfile parameter as the path to the hostfile file, and then run the following command to start multi-machine training.

hostfile="/path/to/hostfile"
deepspeed --hostfile=$hostfile finetune/finetune.py  \
    --report_to "none" \
    --data_path "finetune/data/coig_10k.json" \
    --model_name_or_path deeplang-ai/LingoWhale-8B \
    --output_dir "output" \
    --model_max_length 2048 \
    --num_train_epochs 4 \
    --per_device_train_batch_size 4 \
    --gradient_accumulation_steps 1 \
    --save_strategy epoch \
    --learning_rate 2e-5 \
    --lr_scheduler_type constant \
    --adam_beta1 0.9 \
    --adam_beta2 0.98 \
    --adam_epsilon 1e-8 \
    --max_grad_norm 1.0 \
    --weight_decay 1e-4 \
    --warmup_ratio 0.0 \
    --logging_steps 1 \
    --gradient_checkpointing True \
    --deepspeed finetune/ds_config.json \
    --bf16 True \
    --tf32 True

Parameter-Efficient Fine-tuning

By using peft, you can easily fine-tune methods such as LoRA and Prefix-Tuning. Currently, the LoRA training method is integrated into the code and can be started by adding --use_lora True.

The checkpoint trained with LoRA can be read and called using the following code:

from peft import AutoPeftModelForCausalLM
model = AutoPeftModelForCausalLM.from_pretrained("output", trust_remote_code=True)

📄 License

The community's use of the LingoWhale-8B model needs to comply with Apache 2.0 and LingoWhale-8B Model License Agreement. If you wish to use this source model or its derivatives for commercial purposes, please fill out the questionnaire for application. After approval, you can use it commercially for free.

⚠️ Important Note

Due to the influence of the model's parameter quantity, inherent problems of large models such as hallucination and relatively weak mathematical calculation ability still exist in the LingoWhale-8B model. Please understand these issues before use and evaluate the possible risks. Future versions of the LingoWhale model will focus on optimizing such problems.

💡 Usage Tip

When fine-tuning the model, you can adjust the parameters according to your own needs. If you need to use flash attention or parameter-efficient fine-tuning methods, make sure to install the corresponding libraries.