Sheared LLaMA 1.3B Pruned

🚀 Sheared-LLaMA

Sheared-LLaMA is a series of models derived from Llama2, offering various pruned and instruction - tuned versions for different research and application needs.

🚀 Quick Start

This README provides information about Sheared-LLaMA models, including their paper, code, model links, and license details.

📚 Documentation

Links

• Paper: https://arxiv.org/pdf/2310.06694.pdf
• Code: https://github.com/princeton-nlp/LLM-Shearing
• Models:
  • Sheared-LLaMA-1.3B
  • Sheared-LLaMA-2.7B
• Pruned Models without Continued Pre-training:
  • Sheared-LLaMA-1.3B-Pruned
  • Sheared-LLaMA-2.7B-Pruned
• Instruction-tuned Models:
  • Sheared-LLaMA-1.3B-ShareGPT
  • Sheared-LLaMA-2.7B-ShareGPT

License

Model Details

Sheared-LLaMA-1.3B-Pruned is the model pruned from meta-llama/Llama-2-7b-hf without continued pre-training. We used roughly 0.4B tokens to perform the pruning experiment. This model could be a good use to study:

• Effective data mixtures for continued pre-training
• Comparisons to other pruning techniques
• Extensive evaluations to understand how pruning affects knowledge and reasoning capabilities of LLMs