Miniplm Qwen 200M

🚀 MinPLM-Qwen-200M

MiniPLM-Qwen-200M is a 200M model based on the Qwen architecture. It is pre-trained from scratch using the MiniPLM knowledge distillation framework, aiming to achieve high - performance text generation.

🚀 Quick Start

The MiniPLM-Qwen-200M model is a 200M model with the Qwen architecture. It is pre - trained from scratch on the Pile using the MiniPLM knowledge distillation framework, with the offcial Qwen1.5 - 1.8B as the teacher model.

We also open - source the pre - training corpus refined by Difference Sampling in MiniPLM for reproducibility.

✨ Features

• High Performance: MiniPLM models achieve better performance given the same computation and scale well across model sizes.

📚 Documentation

Baseline Models

• Conventional Pre - Training
• VanillaKD

📄 License

This project is licensed under the Apache 2.0 license.

📖 Citation

@article{miniplm,
    title={MiniPLM: Knowledge Distillation for Pre-Training Language Models}, 
    author={Yuxian Gu and Hao Zhou and Fandong Meng and Jie Zhou and Minlie Huang},
    journal={arXiv preprint arXiv:2410.17215},
    year={2024}
}