Metamath 7B V1.0

🚀 MetaMath-Llemma-7B

MetaMath-Llemma-7B is a model fine - tuned on MetaMathQA datasets, based on the Llemma - 7B model, which significantly improves performance on mathematical tasks.

🚀 Quick Start

Check out our paper at https://arxiv.org/abs/2309.12284. View the project page: https://meta-math.github.io/

✨ Features

• All MetaMathQA data are augmented from the training sets of GSM8K and MATH, with none of the augmented data from the testing set.
• MetaMath-Llemma-7B is fully fine - tuned on the MetaMathQA datasets and based on the powerful Llemma - 7B model, boosting the MATH Pass@1 from 19.8 to 30.0.

📦 Installation

pip install transformers==4.35.0
pip install torch==2.0.1
pip install sentencepiece==0.1.99
pip install tokenizers==0.13.3
pip install accelerate==0.21.0
pip install bitsandbytes==0.40.0
pip install vllm
pip install fraction
pip install protobuf

💻 Usage Examples

Basic Usage

The prompting template is as follows:

"Below is an instruction that describes a task. "
"Write a response that appropriately completes the request.\n\n"
"### Instruction:\n{instruction}\n\n### Response: Let's think step by step."

You need to use your query question to replace the {instruction}.

📚 Documentation

Important Note

All MetaMathQA data are augmented from the training sets of GSM8K and MATH. None of the augmented data is from the testing set. You can check the original_question in meta-math/MetaMathQA, each item is from the GSM8K or MATH train set.

Experiments

Information Table

📄 License

The model is under the llama2 license.

🔧 Technical Details

MetaMath-Llemma-7B is fully fine - tuned on the MetaMathQA datasets and based on the powerful Llemma - 7B model. It is glad to see using MetaMathQA datasets and changing the base model from llama - 2 - 7B to Llemma - 7B can boost the MATH performance from 19.8 to 30.0.

📄 Citation

@article{yu2023metamath,
  title={MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models},
  author={Yu, Longhui and Jiang, Weisen and Shi, Han and Yu, Jincheng and Liu, Zhengying and Zhang, Yu and Kwok, James T and Li, Zhenguo and Weller, Adrian and Liu, Weiyang},
  journal={arXiv preprint arXiv:2309.12284},
  year={2023}
}