🚀 SEA-LION-v1-3B
SEA-LION is a collection of Large Language Models (LLMs) pretrained and instruct - tuned for the Southeast Asia (SEA) region. The model sizes range from 3 billion to 7 billion parameters. This is the card for SEA-LION-v1-3B, which stands for Southeast Asian Languages In One Network.
🚀 Quick Start
This section provides an overview of the SEA-LION-v1-3B model, including its details, training information, and more.
✨ Features
- Regional Focus: Specially trained to understand the Southeast Asian regional context.
- Multilingual Support: Supports multiple languages including English, Chinese, Indonesian, Malay, Thai, Vietnamese, Filipino, Tamil, Burmese, Khmer, and Lao.
- Custom Tokenizer: Employs a custom SEABPETokenizer for optimal performance on SEA languages.
📦 Installation
No installation steps are provided in the original document, so this section is skipped.
💻 Usage Examples
No code examples are provided in the original document, so this section is skipped.
📚 Documentation
Model Details
Model Description
The SEA-LION model represents a significant advancement in Natural Language Processing, specifically trained to understand the SEA regional context. SEA-LION-v1-3B is built on the robust MPT architecture with a vocabulary size of 256K. For tokenization, it uses our custom SEABPETokenizer, tailored for SEA languages to ensure optimal performance.
Property |
Details |
Developed by |
Products Pillar, AI Singapore |
Funded by |
Singapore NRF |
Model Type |
Decoder |
Languages |
English, Chinese, Indonesian, Malay, Thai, Vietnamese, Filipino, Tamil, Burmese, Khmer, Lao |
License |
MIT License |
Performance Benchmarks
SEA-LION-v1-3B has an average performance on general tasks in English (as measured by Hugging Face's LLM Leaderboard):
Model |
ARC |
HellaSwag |
MMLU |
TruthfulQA |
Average |
SEA-LION 3B |
36.26 |
64.59 |
24.07 |
36.46 |
40.35 |
Training Details
Data
SEA-LION-v1-3B was trained on 980B tokens of the following data:
Data Source |
Unique Tokens |
Multiplier |
Total Tokens |
Percentage |
RefinedWeb - English |
571.3B |
1 |
571.3B |
58.20% |
mC4 - Chinese |
91.2B |
1 |
91.2B |
9.29% |
mC4 - Indonesian |
3.68B |
4 |
14.7B |
1.50% |
mC4 - Malay |
0.72B |
4 |
2.9B |
0.29% |
mC4 - Filipino |
1.32B |
4 |
5.3B |
0.54% |
mC4 - Burmese |
1.2B |
4 |
4.9B |
0.49% |
mC4 - Vietnamese |
63.4B |
1 |
63.4B |
6.46% |
mC4 - Thai |
5.8B |
2 |
11.6B |
1.18% |
WangChanBERTa - Thai |
5B |
2 |
10B |
1.02% |
mC4 - Lao |
0.27B |
4 |
1.1B |
0.12% |
mC4 - Khmer |
0.97B |
4 |
3.9B |
0.40% |
mC4 - Tamil |
2.55B |
4 |
10.2B |
1.04% |
the Stack - Python |
20.9B |
2 |
41.8B |
4.26% |
the Stack - Javascript |
55.6B |
1 |
55.6B |
5.66% |
the Stack - Shell |
1.2B5 |
2 |
2.5B |
0.26% |
the Stack - SQL |
6.4B |
2 |
12.8B |
1.31% |
the Stack - Markdown |
26.6B |
1 |
26.6B |
2.71% |
RedPajama - StackExchange |
21.2B |
1 |
21.2B |
2.16% |
RedPajama - ArXiv |
30.6B |
1 |
30.6B |
3.12% |
Infrastructure
SEA-LION-v1-3B was trained using MosaicML Composer on the following hardware:
Training Details |
SEA-LION-v1-3B |
AWS EC2 p4d.24xlarge |
30 instances |
Nvidia A100 40GB GPU |
240 |
Training Duration |
14 days |
Configuration
HyperParameter |
SEA-LION-v1-3B |
Precision |
bfloat16 |
Optimizer |
decoupled_adamw |
Scheduler |
cosine_with_warmup |
Learning Rate |
1.6e-4 |
Global Batch Size |
1200 |
Micro Batch Size |
5 |
Technical Specifications
Model Architecture and Objective
SEA-LION-v1-3B is a decoder model using the MPT architecture.
Parameter |
SEA-LION-v1-3B |
Layers |
32 |
d_model |
2560 |
head_dim |
20 |
Vocabulary |
256000 |
Sequence Length |
2048 |
Tokenizer Details
We sample 20M lines from the training data to train the tokenizer. The framework for training is SentencePiece. The tokenizer type is Byte - Pair Encoding (BPE).
The Team
Lam Wen Zhi Clarence
Leong Wei Qi
Li Yier
Liu Bing Jie Darius
Lovenia Holy
Montalan Jann Railey
Ng Boon Cheong Raymond
Ngui Jian Gang
Nguyen Thanh Ngan
Ong Tat - Wee David
Rengarajan Hamsawardhini
Susanto Yosephine
Tai Ngee Chia
Tan Choon Meng
Teo Jin Howe
Teo Eng Sipp Leslie
Teo Wei Yi
Tjhi William
Yeo Yeow Tong
Yong Xianbin
Acknowledgements
AI Singapore is a national programme supported by the National Research Foundation, Singapore and hosted by the National University of Singapore. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not reflect the views of National Research Foundation, Singapore.
Contact
For more info, please contact us using this SEA-LION Inquiry Form
Link to SEA-LION's GitHub repository
Disclaimer
This the repository for the base model. The model has not been aligned for safety. Developers and users should perform their own safety fine - tuning and related security measures. In no event shall the authors be held liable for any claim, damages, or other liability arising from the use of the released weights and codes.
References
Thai Pre - Training Data Reference
@misc{lowphansirikul2021wangchanberta,
title={WangchanBERTa: Pretraining transformer-based Thai Language Models},
author={Lalita Lowphansirikul and Charin Polpanumas and Nawat Jantrakulchai and Sarana Nutanong},
year={2021},
eprint={2101.09635},
archivePrefix={arXiv},
primaryClass={cs.CL}
}