BELLE 7B 2M

🚀 Model Card for Model ID

BELLE is a large language model based on Bloomz-7b1-mt. It is finetuned with a combination of 2 million Chinese data and 50,000 pieces of English data from Stanford-Alpaca, enabling it to have excellent Chinese instruction understanding and response generation capabilities.

🚀 Quick Start

If you find this model helpful, please like this model and star us on GitHub!

✨ Features

• Multilingual Capability: Supports both Chinese and English, suitable for a wide range of language tasks.
• Instruction Understanding: Can accurately understand various instructions and generate appropriate responses.
• Multiple Model Versions: Trained on datasets of different sizes, providing users with more choices.

📦 Installation

This section is skipped as there is no specific installation step provided in the original document.

💻 Usage Examples

Basic Usage

Please note that the input should be formatted as follows in both training and inference:

Human: {input} \n\nAssistant:

BELLE can be easily loaded with AutoModelForCausalLM.

from transformers import AutoTokenizer, AutoModelForCausalLM
import sys

model_path = "./" # You can modify the path for storing the local model
model =  AutoModelForCausalLM.from_pretrained(model_path)
tokenizer = AutoTokenizer.from_pretrained(model_path)
print("Human:")
line = input()
while line:
        inputs = 'Human: ' + line.strip() + '\n\nAssistant:'
        input_ids = tokenizer(inputs, return_tensors="pt").input_ids
        outputs = model.generate(input_ids, max_new_tokens=200, do_sample = True, top_k = 30, top_p = 0.85, temperature = 0.35, repetition_penalty=1.2)
        rets = tokenizer.batch_decode(outputs, skip_special_tokens=True)
        print("Assistant:\n" + rets[0].strip().replace(inputs, ""))
        print("\n------------------------------------------------\nHuman:")
        line = input()

Advanced Usage

The above example did not utilize the context from previous turns of the conversation. If you want to perform multi-turn dialogue understanding, you can concatenate the previous turns of the conversation in the format of Human:{input}\n\nAssistant:{output} and input it to the generate function.

📚 Documentation

Model description

BELLE is based on Bloomz-7b1-mt and finetuned with 2M Chinese data combined with 50,000 pieces of English data from the open source Stanford-Alpaca, resulting in good Chinese instruction understanding and response generation capabilities.

The code of Chinese data generation and other detailed information can be found in our Github project repository.

We trained models using datasets of different sizes (200,000, 600,000, 1,000,000, and 2,000,000 samples) for instruction learning, and we obtained different model versions as shown below:

Training hyper-parameters

Some use cases

🔧 Technical Details

This section is skipped as there is no specific technical detail provided in the original document.

📄 License

The model is licensed under the Apache-2.0 license.

Citation

Please cite us when using our code, data or model.

@misc{BELLE,
  author = {Yunjie Ji, Yong Deng, Yan Gong, Yiping Peng, Qiang Niu, Baochang Ma, Xiangang Li},
  title = {BELLE: Bloom-Enhanced Large Language model Engine },
  year = {2023},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {\url{https://github.com/LianjiaTech/BELLE}},
}

Also, please cite the original BLOOM, Stanford Alpaca, and Self-Instruct papers.

⚠️ Important Note

There still exist a few issues in the model trained on current base model and data:

1. The model might generate factual errors when asked to follow instructions related to facts.
2. Occasionally generates harmful responses since the model still struggles to identify potential harmful instructions.
3. Needs improvements on reasoning and coding.

Since the model still has its limitations, we require developers only use the open-sourced code, data, model and any other artifacts generated via this project for research purposes. Commercial use and other potential harmful use cases are not allowed.