Mobilellama 1.4B Base

🚀 MobileLLaMA-1.4B-Base

MobileLLaMA-1.4B-Base is a Transformer model with 1.4 billion parameters, downscaled from LLaMA for easy deployment. It is trained on 1.3T tokens from the RedPajama v1 dataset, enabling controlled experiments and showing comparable performance to recent open - source models on benchmarks.

🚀 Quick Start

Model weights can be loaded with Hugging Face Transformers. Examples can be found at Github.

✨ Features

• Downscaled Design: MobileLLaMA-1.4B-Base downscales LLaMA to facilitate off - the - shelf deployment.
• Reproducible Training: All models are trained on 1.3T tokens from the RedPajama v1 dataset only, enabling controlled experiments for further research.
• Benchmark Performance: The model is extensively assessed on two standard natural language benchmarks for language understanding and common sense reasoning, and experimental results show it is on par with the most recent open - source models.

📚 Documentation

Model Summery

MobileLLaMA-1.4B-Base is a Transformer with 1.4 billion parameters. We downscale LLaMA to facilitate the off - the - shelf deployment. To make our work reproducible, all the models are trained on 1.3T tokens from the RedPajama v1 dataset only. This benefits further research by enabling controlled experiments.

We extensively assess our models on two standard natural language benchmarks, for language understanding and common sense reasoning respectively. Experimental results show that our MobileLLaMA 1.4B is on par with the most recent open - source models.

Model Sources

• Repository: https://github.com/Meituan-AutoML/MobileVLM
• Paper: https://arxiv.org/abs/2312.16886

Training Details

Please refer to our paper in section 4.1: MobileVLM: A Fast, Strong and Open Vision Language Assistant for Mobile Devices.

📄 License

The model is released under the Apache 2.0 license.