Finance Llama3 8B

🚀 Instruction Pre-Training: Language Models are Supervised Multitask Learners (EMNLP 2024)

This repository houses the finance model developed from Llama3-8B presented in our paper Instruction Pre-Training: Language Models are Supervised Multitask Learners. We introduce Instruction Pre-Training, a framework for supervised multitask pre-training. It enriches large raw corpora with instruction - response pairs to pre - train language models. These pairs are generated by an efficient instruction synthesizer based on open - source models. Instruction Pre-Training outperforms Vanilla Pre-training in both general pre-training from scratch and domain - adaptive continual pre-training. In pre-training from scratch, it not only enhances pre - trained base models but also benefits more from further instruction tuning. In continual pre - training, it enables Llama3 - 8B to match or even surpass Llama3 - 70B.

✨ Features

Updates

• 2024/11/30: Released the multimodal version of the instruction synthesizer: Visual Instruction Synthesizer
• 2024/9/20: Our paper has been accepted by EMNLP 2024 main conference🎉
• 2024/9/11: Updated FAQ on continual pre-training from Llama3
• 2024/8/29: Updated guidelines on evaluating any 🤗Huggingface models on the domain-specific tasks
• 2024/7/31: Updated pre-training suggestions in the Advanced Usage section of instruction-synthesizer
• 2024/7/15: We scaled up the pre-trained tokens from 100B to 250B, with the number of synthesized instruction-response pairs reaching 500M. The performance trend on downstream tasks throughout the pre-training process:

Resources

🤗 We share our data and models with example usages, feel free to open any discussions at this page! 🤗

• Thanks to the demo davanstrien/instruction-synthesizer for implementing our approach
• Context-Based Instruction Synthesizer: instruction-synthesizer
• Fine-Tuning Data for the Synthesizer: ft-instruction-synthesizer-collection
• General Models Pre-Trained from Scratch (on 100B tokes):
  • InstructLM-500M
  • InstructLM-1.3B
• Domain-Specific Models Pre-Trained from Llama3-8B:
  • Finance-Llama3-8B
  • Biomedicine-Llama3-8B
• General Instruction-Augmented Corpora: general-instruction-augmented-corpora
• Domain-Specific Instruction-Augmented Corpora (no finance data to avoid ethical issues): medicine-instruction-augmented-corpora

Domain-Adaptive Continued Pre-Training

Following AdaptLLM, we augment the domain-specific raw corpora with instruction-response pairs generated by our context-based instruction synthesizer.

💻 Usage Examples

Basic Usage

Chat with the finance-Llama3-8B model

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("instruction-pretrain/finance-Llama3-8B")
tokenizer = AutoTokenizer.from_pretrained("instruction-pretrain/finance-Llama3-8B")

# Put your input here, NO prompt template is required
user_input = '''Use this fact to answer the question: Title of each class Trading Symbol(s) Name of each exchange on which registered
Common Stock, Par Value $.01 Per Share MMM New York Stock Exchange
MMM Chicago Stock Exchange, Inc.
1.500% Notes due 2026 MMM26 New York Stock Exchange
1.750% Notes due 2030 MMM30 New York Stock Exchange
1.500% Notes due 2031 MMM31 New York Stock Exchange

Which debt securities are registered to trade on a national securities exchange under 3M's name as of Q2 of 2023?'''

inputs = tokenizer(user_input, return_tensors="pt", add_special_tokens=True).input_ids.to(model.device)
outputs = model.generate(input_ids=inputs, max_new_tokens=400)[0]

answer_start = int(inputs.shape[-1])
pred = tokenizer.decode(outputs[answer_start:], skip_special_tokens=True)

print(pred)

Advanced Usage

Evaluate any Huggingface LMs on domain-specific tasks (💡New!)

You can use the following script to reproduce our results and evaluate any other Huggingface models on domain-specific tasks. Note that the script is NOT applicable to models that require specific prompt templates (e.g., Llama2-chat, Llama3-Instruct).

1). Set Up Dependencies

git clone https://github.com/microsoft/LMOps
cd LMOps/adaptllm
pip install -r requirements.txt

2). Evaluate the Model

# Select the domain from ['biomedicine', 'finance']
DOMAIN='finance'
  
# Specify any Huggingface LM name (Not applicable to models requiring specific prompt templates)
MODEL='instruction-pretrain/finance-Llama3-8B'
  
# Model parallelization:
# - Set MODEL_PARALLEL=False if the model fits on a single GPU. 
#   We observe that LMs smaller than 10B always meet this requirement.
# - Set MODEL_PARALLEL=True if the model is too large and encounters OOM on a single GPU.
MODEL_PARALLEL=False
  
# Choose the number of GPUs from [1, 2, 4, 8]
N_GPU=1
  
# Whether to add a BOS token at the beginning of the prompt input:
# - Set to False for AdaptLLM.
# - Set to True for instruction-pretrain models.
# If unsure, we recommend setting it to False, as this is suitable for most LMs.
add_bos_token=True

# Run the evaluation script
bash scripts/inference.sh ${DOMAIN} ${MODEL} ${add_bos_token} ${MODEL_PARALLEL} ${N_GPU}

📚 Documentation

FAQ on Continual Pre-Training from LLama3

Q1: Do you use the official Llama3 instruction prompt for pre-training?

No, the provided Llama3 instruction prompt is designed for the instruction-tuned model, but our continual pre-training is conducted on the pre-trained base model where only BOS (<|begin_of_text|>) and EOS (<|end_of_text|>) tokens are required.

Q2: For the general instructions from OpenOrca, do you concatenate each instruction with its output using '\n'?

No, as mentioned in the pre-training suggestions, we use a simple whitespace to concatenate each question with its response for the general instruction data from OpenOrca. This is because OpenOrca's data is already templated with diverse natural languge templates (such as those with \n), so a whitespace is sufficient to formulate the data.

Note that when using our templated instruction-augmented texts, you don't need to add any concatenations.

Q3: What about those system prompts in OpenOrca?

We simply discard the system prompts.

To put it all together, the text before tokenization looks like this:

general_instruction_response_text = "<|begin_of_text|>{question} {response}<|end_of_text|>"

instruction_augmented_text = "<|begin_of_text|>{instruction augmented text}<|end_of_text|>"

Then, for tokenization, you don't need to add BOS and EOS token ids. The tokenization code looks like this:

text_ids = tokenizer(text, add_special_tokens=False, **kwargs).input_ids

📄 License

The license for this project is llama3.

📚 Documentation

Citation

If you find our work helpful, please cite us:

Instruction Pre-Training (EMNLP 2024)

@article{cheng2024instruction,
  title={Instruction Pre-Training: Language Models are Supervised Multitask Learners},
  author={Cheng, Daixuan and Gu, Yuxian and Huang, Shaohan and Bi, Junyu and Huang, Minlie and Wei, Furu},
  journal={arXiv preprint arXiv:2406.14491},
  year={2024}
}

Adapt LLM to Domains (ICLR 2024)

@inproceedings{
cheng2024adapting,
title={Adapting Large Language Models via Reading Comprehension},
author={Daixuan Cheng and Shaohan Huang and Furu Wei},
booktitle={The Twelfth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=y886UXPEZ0}
}