Metamath Mistral 7B

🚀 MetaMath-Mistral-7B

MetaMath-Mistral-7B is a fully fine - tuned model on the MetaMathQA datasets, leveraging the powerful Mistral - 7B architecture. It significantly enhances performance on GSM8K, demonstrating the effectiveness of the MetaMathQA datasets.

🚀 Quick Start

• See our paper: https://arxiv.org/abs/2309.12284
• View the project page: [https://meta - math.github.io/](https://meta - math.github.io/)

✨ Features

• Data Augmentation: All MetaMathQA data are augmented from the training sets of GSM8K and MATH, with none from the testing set.
• Performance Boost: Changing the base model from llama - 2 - 7B to Mistral - 7B and using MetaMathQA datasets boosts GSM8K performance from 66.5 to 77.7.

📦 Installation

pip install transformers==4.35.0
pip install torch==2.0.1
pip install sentencepiece==0.1.99
pip install tokenizers==0.13.3
pip install accelerate==0.21.0
pip install bitsandbytes==0.40.0
pip install vllm
pip install fraction
pip install protobuf

💻 Usage Examples

Basic Usage

The prompting template is as follows:

prompt = '''
"Below is an instruction that describes a task. "
"Write a response that appropriately completes the request.\n\n"
"### Instruction:\n{instruction}\n\n### Response: Let's think step by step."
'''
# Replace {instruction} with your query question

Advanced Usage

There is another interesting repo about Arithmo - Mistral - 7B at [https://huggingface.co/akjindal53244/Arithmo - Mistral - 7B](https://huggingface.co/akjindal53244/Arithmo - Mistral - 7B), which combines our MetaMathQA dataset and MathInstruct datasets to train a powerful model. We also plan to train on the combination of MetaMathQA and MathInstruct datasets and open all results and training details.

📚 Documentation

Note

All MetaMathQA data are augmented from the training sets of GSM8K and MATH. None of the augmented data is from the testing set. You can check the original_question in meta - math/MetaMathQA, each item is from the GSM8K or MATH train set.

Model Details

MetaMath - Mistral - 7B is fully fine - tuned on the MetaMathQA datasets and based on the Mistral - 7B model. To fine - tune Mistral - 7B, it is recommended to use a smaller learning rate (usually 1/5 to 1/10 of the lr for LlaMa - 2 - 7B) while keeping other training args unchanged. More training details and scripts can be found at [https://github.com/meta - math/MetaMath](https://github.com/meta - math/MetaMath).

Experiments

🔧 Technical Details

To fine - tune Mistral - 7B, use a smaller learning rate (usually 1/5 to 1/10 of the lr for LlaMa - 2 - 7B) and keep other training args unchanged. More details are available at [https://github.com/meta - math/MetaMath](https://github.com/meta - math/MetaMath).

📄 License

This project is under the Apache - 2.0 license.

📖 Citation

@article{yu2023metamath,
  title={MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models},
  author={Yu, Longhui and Jiang, Weisen and Shi, Han and Yu, Jincheng and Liu, Zhengying and Zhang, Yu and Kwok, James T and Li, Zhenguo and Weller, Adrian and Liu, Weiyang},
  journal={arXiv preprint arXiv:2309.12284},
  year={2023}
}

@article{jiang2023mistral,
  title={Mistral 7B},
  author={Jiang, Albert Q and Sablayrolles, Alexandre and Mensch, Arthur and Bamford, Chris and Chaplot, Devendra Singh and Casas, Diego de las and Bressand, Florian and Lengyel, Gianna and Lample, Guillaume and Saulnier, Lucile and others},
  journal={arXiv preprint arXiv:2310.06825},
  year={2023}
}