The open-source model gpt3-finnish-small - Supports Finnish text generation. Experience intelligent content creation for free.

Gpt3 Finnish Small

Developed by TurkuNLP

This is a Finnish generative pretrained Transformer model with 186 million parameters, based on the BLOOM architecture and trained on 300 billion tokens.

Large Language Model

Transformers

Open Source License:Apache-2.0 #Finnish monolingual model #BLOOM architecture #Base model can be fine-tuned

Downloads 910

Release Time : 2/15/2023

Model Overview

The TurkuNLP Finnish GPT-3 series models are a family of pretrained monolingual GPT-style language models based on the BLOOM architecture and can be used as base models for further instruction fine-tuning.

Model Features

Based on the BLOOM architecture

A monolingual GPT-style language model using the BLOOM architecture, with efficient text generation capabilities.

Large-scale training data

Trained on 300 billion tokens, covering a variety of Finnish language resources.

Can be used as a base model

Suitable for further instruction fine-tuning, such as being used as a modern chat model.

Model Capabilities

Finnish text generation

Language model pretraining

Foundation for instruction fine-tuning

Use Cases

Natural language processing

Finnish text generation

Generate coherent Finnish text, suitable for scenarios such as content creation and automatic replies.

Foundation for instruction fine-tuning

As a base model, it can be used for chatbots or other dialogue systems through instruction fine-tuning.

🚀 Finnish Generative Pretrained Transformer

This is a Generative Pretrained Transformer with 186M parameters designed for the Finnish language. TurkuNLP Finnish GPT - 3 - models are a family of pretrained monolingual GPT - style language models based on the BLOOM - architecture.

⚠️ Important Note Note that the models are pure language models, meaning that they are not instruction finetuned for dialogue or answering questions. These models are intended to be used as foundational models that can be, for example, instruction finetuned to serve as modern chat - models. All models are trained for 300B tokens.

✨ Features

Based on the BLOOM - architecture, providing a solid foundation for Finnish language processing.
Can be used as a starting point for various downstream tasks, such as instruction finetuning for chat - models.

📦 Model Parameters

Model	Layers	Dim	Heads	Params
Small	12	768	12	186M
Medium	24	1024	16	437M
Large	24	1536	16	881M
XL	24	2064	24	1.5B
”3B”	32	2560	32	2.8B
”8B”	32	4096	32	7.5B
"13B"	40	5120	40	13.3B

📚 Datasets

The models are trained using a combination of multiple Finnish resources:

Finnish Internet Parsebank: https://turkunlp.org/finnish_nlp.html
mC4 multilingual colossal, cleaned Common Crawl: https://huggingface.co/datasets/mc4
Common Crawl Finnish: https://TODO
Finnish Wikipedia: https://fi.wikipedia.org/wiki
Lönnrot Projekti Lönnrot: http://www.lonnrot.net/
ePub National library ”epub” collection
National library ”lehdet” collection
Suomi24 The Suomi 24 Corpus 2001 - 2020: http://urn.fi/urn:nbn:fi:lb-2021101527
Reddit r/Suomi submissions and comments: https://www.reddit.com/r/Suomi
STT Finnish News Agency Archive 1992 - 2018: http://urn.fi/urn:nbn:fi:lb-2019041501
Yle Finnish News Archive 2011 - 2018: http://urn.fi/urn:nbn:fi:lb-2017070501
Yle Finnish News Archive 2019 - 2020: http://urn.fi/urn:nbn:fi:lb-2021050401
ROOTS: TODO

🔧 Sampling Ratios

Dataset	Chars	Ratio	Weight	W.Ratio
Parsebank	35.0B	16.9%	1.5	22.7%
mC4 - Fi	46.3B	22.4%	1.0	20.0%
CC - Fi	79.6B	38.5%	1.0	34.4%
Fiwiki	0.8B	0.4%	3.0	1.0%
Lönnrot	0.8B	0.4%	3.0	1.0%
Yle	1.6B	0.8%	2.0	1.4%
STT	2.2B	1.1%	2.0	1.9%
ePub	13.5B	6.5%	1.0	5.8%
Lehdet	5.8B	2.8%	1.0	2.5%
Suomi24	20.6B	9.9%	1.0	8.9%
Reddit - Fi	0.7B	0.4%	1.0	0.3%
TOTAL	207.0B	100.0%	N/A	100.0%

📄 License

The models are released under the apache - 2.0 license.

📢 More documentation and a paper are coming soon.

Featured Recommended AI Models

Empowering the Future, Your AI Solution Knowledge Base

English 简体中文繁體中文にほんご