Zhilu 13B Instruct

🚀 ZhiLu-13B-Instruct

ZhiLu is a large financial model fine - tuned based on Chinese Alpaca2-13B. We conducted incremental pre - training using a large amount of Chinese and English corpora and aligned it with high - quality instruction data.

The goal of model training is to significantly enhance the capabilities in the financial domain while maintaining general capabilities. For specific details, refer to the ZhiLu GitHub repository.

🚀 Quick Start

This repository provides the complete ZhiLu model. Users don't need to download the LoRA module separately when using this model.

✨ Features

Data Quality

We collected various types of high - quality data, including announcements of listed companies, financial news, annual reports of listed companies, general news, financial information, community Q&A, and Wikipedia.

The total number of tokens for model training is 14.69B. The ratio of general corpora to financial corpora is approximately 2:1, and the ratio of Chinese to English data is also about 2:1.

Model Training

ZhiLu uses LoRA for efficient training (including emb/lm - head) and FlashAttention - 2 technology for accelerated training. We have released the Full Model (this repository) and the LoRA module.

💻 Usage Examples

Basic Usage

import torch
from transformers import LlamaForCausalLM, LlamaTokenizer
model_name_or_path = ""
tokenizer = LlamaTokenizer.from_pretrained(model_name_or_path, use_fast=False, legacy=True)
model = LlamaForCausalLM.from_pretrained(model_name_or_path, torch_dtype=torch.bfloat16,device_map="auto")
inputs = tokenizer("什么是A股？", return_tensors="pt").to("cuda")
outputs = model.generate(**inputs, max_new_tokens=64, repetition_penalty=1.1)
outputs = tokenizer.decode(outputs.cpu()[0][len(inputs.input_ids[0]):], skip_special_tokens=True)
print(outputs)

📄 License

The model is licensed under the Apache 2.0 license.