🚀 TurkuNLP Finnish GPT-3 Models
A family of Finnish text generation models based on the BLOOM architecture, offering various parameter sizes for different needs.
🚀 Quick Start
The TurkuNLP Finnish GPT - 3 models are a family of pretrained monolingual GPT - style language models based on the BLOOM architecture. These models have 1.5B parameters for Finnish text generation. Note that these are pure language models and not instruction finetuned for dialogue or answering questions. They are designed to be used as foundational models, which can be further instruction - finetuned to serve as modern chat - models.
✨ Features
- Multiple Sizes: Available in different parameter sizes (from 186M to 13.3B) to suit various computational and performance requirements.
- Finnish - Focused: Specifically trained for the Finnish language, using a combination of multiple Finnish resources.
- Large Training Data: All models are trained on 300B tokens.
📚 Documentation
📦 Model Parameters
| Model |
Layers |
Dim |
Heads |
Params |
| Small |
12 |
768 |
12 |
186M |
| Medium |
24 |
1024 |
16 |
437M |
| Large |
24 |
1536 |
16 |
881M |
| XL |
24 |
2064 |
24 |
1.5B |
| ”3B” |
32 |
2560 |
32 |
2.8B |
| ”8B” |
32 |
4096 |
32 |
7.5B |
| "13B" |
40 |
5120 |
40 |
13.3B |
📊 Datasets
We used a combination of multiple Finnish resources for training:
- Finnish Internet Parsebank https://turkunlp.org/finnish_nlp.html
- mC4 multilingual colossal, cleaned Common Crawl https://huggingface.co/datasets/mc4
- Common Crawl Finnish https://TODO
- Finnish Wikipedia https://fi.wikipedia.org/wiki
- Lönnrot Projekti Lönnrot http://www.lonnrot.net/
- ePub National library ”epub” collection
- National library ”lehdet” collection
- Suomi24 The Suomi 24 Corpus 2001 - 2020 http://urn.fi/urn:nbn:fi:lb-2021101527
- Reddit r/Suomi submissions and comments https://www.reddit.com/r/Suomi
- STT Finnish News Agency Archive 1992 - 2018 http://urn.fi/urn:nbn:fi:lb-2019041501
- Yle Finnish News Archive 2011 - 2018 http://urn.fi/urn:nbn:fi:lb-2017070501
- Yle Finnish News Archive 2019 - 2020 http://urn.fi/urn:nbn:fi:lb-2021050401
- ROOTS TODO
📈 Sampling Ratios
| Dataset |
Chars |
Ratio |
Weight |
W.Ratio |
| Parsebank |
35.0B |
16.9% |
1.5 |
22.7% |
| mC4 - Fi |
46.3B |
22.4% |
1.0 |
20.0% |
| CC - Fi |
79.6B |
38.5% |
1.0 |
34.4% |
| Fiwiki |
0.8B |
0.4% |
3.0 |
1.0% |
| Lönnrot |
0.8B |
0.4% |
3.0 |
1.0% |
| Yle |
1.6B |
0.8% |
2.0 |
1.4% |
| STT |
2.2B |
1.1% |
2.0 |
1.9% |
| ePub |
13.5B |
6.5% |
1.0 |
5.8% |
| Lehdet |
5.8B |
2.8% |
1.0 |
2.5% |
| Suomi24 |
20.6B |
9.9% |
1.0 |
8.9% |
| Reddit - Fi |
0.7B |
0.4% |
1.0 |
0.3% |
| TOTAL |
207.0B |
100.0% |
N/A |
100.0% |
⚠️ Important Note
The models are pure language models and not instruction finetuned for dialogue or answering questions.
📄 License
This project is licensed under the Apache - 2.0 license.
📢 More documentation and a paper are coming soon.
%20--%3e%3cdefs%3e%3cstyle%3e%20.st0%20{%20fill:%20%23061b40;%20}%20.st1%20{%20fill:%20%23306af1;%20}%20.st2%20{%20fill:%20%235ce5cf;%20}%20%3c/style%3e%3c/defs%3e%3cg%3e%3cpath%20class='st0'%20d='M55,10.5h9v3h-9v9h12V7.5h-12v3ZM64,19.5h-6v-3h6v3Z'/%3e%3cpolygon%20class='st0'%20points='69%2016.5%2078%2016.5%2078%2019.5%2069%2019.5%2069%2022.5%2081%2022.5%2081%2013.5%2072%2013.5%2072%2010.5%2081%2010.5%2081%207.5%2069%207.5%2069%2016.5'/%3e%3cpolygon%20class='st0'%20points='95%2010.5%2095%207.5%2083%207.5%2083%2022.5%2095%2022.5%2095%2019.5%2086%2019.5%2086%2016.5%2095%2016.5%2095%2013.5%2086%2013.5%2086%2010.5%2095%2010.5'/%3e%3cpath%20class='st0'%20d='M40,1.5v21h11.6l1.4-1.4v-7.6h0c0,0-1.4-1.5-1.4-1.5l1.4-1.4V2.9l-1.4-1.4h-11.6ZM50,19.5h-7v-6h7v6ZM50,10.5h-7v-6h7v6Z'/%3e%3c/g%3e%3cpath%20class='st1'%20d='M23.1,24L14.7,4.8l-4.9,11.2h3.8l-1.8,4H3.3L12.1,0H2C.9,0,0,.9,0,2v20c0,1.1.9,2,2,2h21.1Z'/%3e%3cpath%20class='st2'%20d='M34,0h-16.8l10.6,24h6.2c1.1,0,2-.9,2-2V2C36,.9,35.1,0,34,0ZM32.5,20h-4V4h4v16Z'/%3e%3c/svg%3e)
