Llama 1B Dj Refine 150B

🚀 Reference LLM from Data-Juicer

This is a reference large language model (LLM) from Data-Juicer, which offers high - performance based on refined datasets.

🚀 Quick Start

For more information about our first data - centric LLM competition, please visit the official websites of FT - Data Ranker (1B Track, 7B Track).

✨ Features

• Model Architecture: The model uses the LLaMA - 1.3B architecture and adopts the OpenLLaMA implementation.
• Training Data: It is pre - trained on 150B tokens of Data - Juicer's refined RedPajama and Pile datasets.
• Performance: It achieves an average score of 34.21 over 16 HELM tasks, outperforming Falcon - 1.3B (trained on 350B tokens from RefinedWeb), Pythia - 1.4B (trained on 300B tokens from the original Pile), and Open - LLaMA - 1.3B (trained on 150B tokens from the original RedPajama and Pile).

📚 Documentation

For more details, please refer to our paper.

📄 License

The model is released under the Apache - 2.0 license.

📦 Training Datasets