SEA LION V1 7B

🚀 SEA-LION-v1-7B

SEA-LION is a collection of Large Language Models (LLMs) pretrained and instruct - tuned for the Southeast Asia (SEA) region. The model sizes range from 3 billion to 7 billion parameters. This README is for the SEA - LION 7B base model. SEA - LION stands for Southeast Asian Languages In One Network.

🚀 Quick Start

SEA-LION is a significant advancement in Natural Language Processing, specifically trained for the SEA regional context. This 7B base model is built on the MPT architecture, with a vocabulary size of 256K and uses a custom SEABPETokenizer for SEA languages.

✨ Features

• Regional Focus: Specially trained for the Southeast Asian context, supporting multiple SEA languages.
• Robust Architecture: Built on the MPT architecture, ensuring stable performance.
• Custom Tokenizer: Uses the SEABPETokenizer, optimized for SEA languages.

📦 Installation

No installation steps are provided in the original document, so this section is skipped.

📚 Documentation

📋 Model Details

Model Description

The SEA - LION model represents a major step forward in Natural Language Processing, trained to understand the SEA regional context. SEA - LION - v1 - 7B is based on the MPT architecture and has a vocabulary size of 256K. It uses the custom SEABPETokenizer for SEA languages to ensure optimal performance. The training data for this model consists of 980B tokens.

Performance Benchmarks

SEA - LION - v1 - 7B shows average performance on general English tasks (as measured by Hugging Face's LLM Leaderboard):

📈 Training Details

Data

SEA - LION - v1 - 7B was trained on 980B tokens from the following data sources:

Infrastructure

SEA - LION - v1 - 7B was trained using MosaicML Composer on the following hardware:

Configuration

🔧 Technical Details

Model Architecture and Objective

SEA - LION - v1 - 7B is a decoder model using the MPT architecture.

Tokenizer Details

We sampled 20M lines from the training data to train the tokenizer. The training framework is SentencePiece, and the tokenizer type is Byte - Pair Encoding (BPE).

👥 The Team

Lam Wen Zhi Clarence
Leong Wei Qi
Li Yier
Liu Bing Jie Darius
Lovenia Holy
Montalan Jann Railey
Ng Boon Cheong Raymond
Ngui Jian Gang
Nguyen Thanh Ngan
Ong Tat - Wee David
Rengarajan Hamsawardhini
Susanto Yosephine
Tai Ngee Chia
Tan Choon Meng
Teo Jin Howe
Teo Eng Sipp Leslie
Teo Wei Yi
Tjhi William
Yeo Yeow Tong
Yong Xianbin

🙏 Acknowledgements

AI Singapore is a national programme supported by the National Research Foundation, Singapore and hosted by the National University of Singapore. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not reflect the views of National Research Foundation, Singapore.

📞 Contact

For more info, please contact us using this SEA - LION Inquiry Form

Link to SEA - LION's GitHub repository

⚠️ Disclaimer

This is the repository for the base model. The model has not been aligned for safety. Developers and users should perform their own safety fine - tuning and related security measures. In no event shall the authors be held liable for any claim, damages, or other liability arising from the use of the released weights and codes.

📚 References

Thai Pre - Training Data Reference

@misc{lowphansirikul2021wangchanberta,
    title={WangchanBERTa: Pretraining transformer-based Thai Language Models},
    author={Lalita Lowphansirikul and Charin Polpanumas and Nawat Jantrakulchai and Sarana Nutanong},
    year={2021},
    eprint={2101.09635},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}