Firefly Bloom 1b4

🚀 Firefly: A Chinese Conversational Large Language Model

Firefly is an open - source Chinese conversational large language model. It is fine - tuned on Chinese datasets using Instruction Tuning. Techniques such as vocabulary pruning, ZeRO, and tensor parallelism are employed to effectively reduce GPU memory consumption and improve training efficiency. During training, we used a smaller number of model parameters and less computing resources.

We constructed a large amount of data related to Chinese culture to enhance the model's performance in this area, including couplets, poetry creation, classical Chinese translation, essays, and Jin Yong's novels.

The name "Firefly" is a symbol of traditional Chinese culture. Although the saying "decayed grass turns into fireflies" has a sense of pity, even a small firefly can light up the night sky with its faint glow. The name of this project is taken from Du Mu's poem "Autumn Eve": "By candlelight in autumn cold the painted screen is seen; With silken fan she chases fireflies on the green." We hope that this project can shine a faint light, just like a firefly, and contribute to the Chinese NLP open - source community.

🚀 Quick Start

The project address is Firefly(流萤): 中文对话式大语言模型. If you find this model helpful, please like it and star our GitHub project.

✨ Features

• Dataset: [firefly - train - 1.1M](https://huggingface.co/datasets/YeungNLP/firefly - train - 1.1M), a high - quality Chinese multi - task instruction fine - tuning dataset containing 1.1M samples. It includes instruction data for 23 common Chinese NLP tasks. For each task, several instruction templates are manually written to ensure the high quality and richness of the data.
• Model Pruning: LLMPruner: A Large Language Model Pruning Tool. We use vocabulary pruning technology to prune the weights of multilingual large language models. While retaining pre - trained knowledge, we effectively reduce the number of model parameters and improve training efficiency. We also share the weights of Bloom models with different parameter scales after pruning.
• Weight Sharing: Based on [bloom - 1b4 - zh](https://huggingface.co/YeungNLP/bloom - 1b4 - zh) and [bloom - 2b6 - zh](https://huggingface.co/YeungNLP/bloom - 2b6 - zh), we perform instruction fine - tuning to obtain two Chinese models with different parameter scales: [firefly - 1b4](https://huggingface.co/YeungNLP/firefly - 1b4) and [firefly - 2b6 (to be shared after training)](https://huggingface.co/YeungNLP/firefly - 2b6).
• Training Code: The training code is open - source, supporting large - model training strategies such as tensor parallelism, ZeRO, and Gemini heterogeneous memory space management. It enables the training of models with 1B - 2B parameters using only one GPU (to be open - sourced after organization).

📦 Installation

No specific installation steps are provided in the original document, so this section is skipped.

💻 Usage Examples

Basic Usage

from transformers import BloomTokenizerFast, BloomForCausalLM
device = 'cuda'
path = 'YeungNLP/firefly-1b4'

tokenizer = BloomTokenizerFast.from_pretrained(path)
model = BloomForCausalLM.from_pretrained(path)
model.eval()
model = model.to(device)
text = input('User：')
while True:
    text = '<s>{}</s></s>'.format(text)
    input_ids = tokenizer(text, return_tensors="pt").input_ids
    input_ids = input_ids.to(device)
    outputs = model.generate(input_ids, max_new_tokens=200, do_sample=True, top_p=0.85, temperature=0.35,
                             repetition_penalty=1.2, eos_token_id=tokenizer.eos_token_id)
    rets = tokenizer.batch_decode(outputs)
    output = rets[0].strip().replace(text, "").replace('</s>', "")
    print("Firefly：{}".format(output))
    text = input('User：')

📚 Documentation

Model Pruning

This project first uses the LLMPruner project to perform vocabulary pruning on the original Bloom model. We only extract the commonly used Chinese and English vocabularies, which significantly reduces the number of model parameters. Then, we perform instruction fine - tuning on it.

Bloom is a multilingual model. Due to the need to be compatible with multiple languages, its vocabulary contains more than 250,000 words. In the Chinese domain, most of the vocabulary is not used. By deleting redundant vocabulary, we extract the commonly used Chinese and English vocabularies from the multilingual model. Finally, the vocabulary is reduced from 250,000 to 46,145, which is 18.39% of the original. While retaining pre - trained knowledge, we effectively reduce the number of parameters and improve training efficiency.

Based on [bloom - 1b4 - zh](https://huggingface.co/YeungNLP/bloom - 1b4 - zh) and [bloom - 2b6 - zh](https://huggingface.co/YeungNLP/bloom - 2b6 - zh), we perform instruction fine - tuning to obtain two Chinese models with different parameter scales: [firefly - 1b4](https://huggingface.co/YeungNLP/firefly - 1b4) and [firefly - 2b6 (to be shared after training)](https://huggingface.co/YeungNLP/firefly - 2b6).

Dataset

We collected 23 common Chinese datasets. For each task, several instruction templates are manually written to ensure the high quality and richness of the data. The dataset contains 1.15 million samples, forming the training set [firefly - train - 1.1M](https://huggingface.co/datasets/YeungNLP/firefly - train - 1.1M). The data distribution is shown in the following figure:

On this basis, we added Belle - train_0.5M_CN, and finally obtained 1.65 million training data. The format of each data entry is as follows, including the task type, input, and target output:

{
  "kind": "ClassicalChinese", 
  "input": "将下面句子翻译成现代文：\n石中央又生一树，高百余尺，条干偃阴为五色，翠叶如盘，花径尺余，色深碧，蕊深红，异香成烟，著物霏霏。",
  "target": "大石的中央长着一棵树，一百多尺高，枝干是彩色的，树叶有盘子那样大，花的直径有一尺宽，花瓣深蓝色，花中飘出奇异的香气笼罩着周围，如烟似雾。"
}

The token length distribution of the training dataset is shown in the following figure. Most of the data has a length of less than 600:

Training Details

During training, for each sample, we process the input and target into the following format and then input it into the model:

<s>input</s></s>target</s>

We set the maximum input length of the model to 512. The input part does not participate in the calculation of the loss, and only the loss function of the target part is calculated. The training hyperparameters are set as shown in the following table:

Limitations and Usage Restrictions

After vocabulary pruning, the number of parameters of our model is only 1.4B and 2.6B, which is far less than models with tens or hundreds of billions of parameters such as ChatGPT and LLaMA, and even far less than current mainstream models such as Belle and ChatGLM with about 7B parameters. Therefore, there are still the following problems in performance:

• It is prone to giving incorrect responses to factual knowledge.
• Since it has not undergone harmless fine - tuning, it may generate discriminatory, harmful, or unethical remarks.
• It still lacks capabilities in code generation and reasoning.

Based on the above limitations of the model, we require that the code, data, and model of this project are only used for academic research, not for commercial purposes, and not for purposes that may cause harm to society.

📄 License

No license information is provided in the original document, so this section is skipped.

🔧 Technical Details

The relevant technical details have been described in the above sections, so this section is not separately presented.

📖 Citation

If you use the data, code, or model of this project, please cite it as follows:

@misc{Firefly,
  author = {Jianxin Yang},
  title = {Firefly(流萤): 中文对话式大语言模型},
  year = {2023},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {\url{https://github.com/yangjianxin1/Firefly}},
}

👀 Follow Us

🎉 Generation Results

The following shows some generation results of firefly - 1b4.