Metamath Llemma 7B

🚀 MetaMath-Llemma-7B

MetaMath-Llemma-7B is a fully fine - tuned model on the MetaMathQA datasets, based on the powerful Llemma - 7B model, aiming to enhance mathematical question - answering capabilities.

🚀 Quick Start

See our paper at https://arxiv.org/abs/2309.12284.

View the project page: [https://meta - math.github.io/](https://meta - math.github.io/)

✨ Features

• All MetaMathQA data are augmented from the training sets of GSM8K and MATH. None of the augmented data is from the testing set.
• MetaMath - Llemma - 7B is fully fine - tuned on the MetaMathQA datasets and based on the Llemma - 7B model, which can boost the MATH performance from 19.8 to 30.0.

📦 Installation

pip install transformers==4.35.0
pip install torch==2.0.1
pip install sentencepiece==0.1.99
pip install tokenizers==0.13.3
pip install accelerate==0.21.0
pip install bitsandbytes==0.40.0
pip install vllm
pip install fraction
pip install protobuf

💻 Usage Examples

Basic Usage

Prompting template:

"Below is an instruction that describes a task. "
"Write a response that appropriately completes the request.\n\n"
"### Instruction:\n{instruction}\n\n### Response: Let's think step by step."

where you need to use your query question to replace the {instruction}.

📚 Documentation

Experiments

📄 License

This project is licensed under the Apache-2.0 license.

📚 Citation

@article{yu2023metamath,
  title={MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models},
  author={Yu, Longhui and Jiang, Weisen and Shi, Han and Yu, Jincheng and Liu, Zhengying and Zhang, Yu and Kwok, James T and Li, Zhenguo and Weller, Adrian and Liu, Weiyang},
  journal={arXiv preprint arXiv:2309.12284},
  year={2023}
}

@article{azerbayev2023llemma,
  title={Llemma: An open language model for mathematics},
  author={Azerbayev, Zhangir and Schoelkopf, Hailey and Paster, Keiran and Santos, Marco Dos and McAleer, Stephen and Jiang, Albert Q and Deng, Jia and Biderman, Stella and Welleck, Sean},
  journal={arXiv preprint arXiv:2310.10631},
  year={2023}
}

⚠️ Important Note

All MetaMathQA data are augmented from the training sets of GSM8K and MATH. None of the augmented data is from the testing set. You can check the original_question in meta-math/MetaMathQA, each item is from the GSM8K or MATH train set.