FinBERT: An Open-Source Finnish Natural Language Processing Model - Optimized Specifically for Finnish, Outperforming Multilingual BERT

Bert Base Finnish Cased V1

Developed by TurkuNLP

FinBERT is the Finnish version of Google's BERT model, specifically optimized for Finnish natural language processing tasks. Trained on large-scale Finnish corpora, it outperforms multilingual BERT in multiple tasks.

Large Language Model Other#Finnish language optimization #Transfer learning #Natural language processing

Downloads 10.30k

Release Time : 3/2/2022

Model Overview

A Finnish pre-trained language model based on the BERT architecture, supporting fine-tuning for various Finnish NLP tasks. Uses a customized vocabulary for more comprehensive Finnish word coverage, trained on diverse corpora including news and forums.

Model Features

Customized vocabulary

Includes 50,000 Finnish-optimized word pieces, significantly improving vocabulary coverage compared to multilingual BERT

Large-scale pre-training

Trained on 3 billion Finnish tokens (24 billion characters), 30 times the size of Finnish Wikipedia

Domain adaptability

Training data covers news, online discussions, and web-crawled content, supporting diverse application scenarios

Model Capabilities

Text classification

Named entity recognition

Part-of-speech tagging

Semantic understanding

Use Cases

News analysis

News topic classification

Automatic classification of Yle news articles

Outperforms multilingual BERT across different training data scales

Social media analysis

Forum content classification

Classification of Ylilauta online discussion content

Significantly higher accuracy than FastText baseline models

Information extraction

Named entity recognition

Identifying Finnish personal names, locations, and other entities from text

Achieved 92.4% accuracy on the FiNER corpus

🚀 FinBERT

A Finnish version of Google's BERT deep transfer learning model, which can be fine - tuned for various Finnish natural language processing tasks and outperforms previous models.

🚀 Quick Start

Release 1.0 (November 25, 2019)

We generally recommend the use of the cased model.

Paper presenting Finnish BERT: arXiv:1912.07076

✨ Features

A version of Google's BERT deep transfer learning model for Finnish. The model can be fine - tuned to achieve state - of - the - art results for various Finnish natural language processing tasks.

FinBERT features a custom 50,000 wordpiece vocabulary that has much better coverage of Finnish words than e.g. the previously released multilingual BERT models from Google:

Property	Details
Vocabulary Coverage	FinBERT has better coverage of Finnish words. For example, in a sentence: FinBERT: Suomessa vaihtuu kesän aikana sekä pääministeri että valtiovarain ##ministeri . Multilingual BERT: Suomessa vai ##htuu kes ##än aikana sekä p ##ää ##minister ##i että valt ##io ##vara ##in ##minister ##i .
Training Data	FinBERT has been pre - trained for 1 million steps on over 3 billion tokens (24B characters) of Finnish text drawn from news, online discussion, and internet crawls. By contrast, Multilingual BERT was trained on Wikipedia texts, where the Finnish Wikipedia text is approximately 3% of the amount used to train FinBERT.

These features allow FinBERT to outperform not only Multilingual BERT but also all previously proposed models when fine - tuned for Finnish natural language processing tasks.

📚 Documentation

Results

Document classification

learning curves for Yle and Ylilauta document classification

FinBERT outperforms multilingual BERT (M - BERT) on document classification over a range of training set sizes on the Yle news (left) and Ylilauta online discussion (right) corpora. (Baseline classification performance with FastText included for reference.)

[code][Yle data] [Ylilauta data]

Named Entity Recognition

Evaluation on FiNER corpus (Ruokolainen et al 2019)

Model	Accuracy
FinBERT	92.40%
Multilingual BERT	90.29%
FiNER - tagger (rule - based)	86.82%

(FiNER tagger results from Ruokolainen et al. 2019)

[code][data]

Part of speech tagging

Evaluation on three Finnish corpora annotated with Universal Dependencies part - of - speech tags: the Turku Dependency Treebank (TDT), FinnTreeBank (FTB), and Parallel UD treebank (PUD)

Model	TDT	FTB	PUD
FinBERT	98.23%	98.39%	98.08%
Multilingual BERT	96.97%	95.87%	97.58%

[code][data]

Previous releases

Release 0.2

October 24, 2019 Beta version of the BERT base uncased model trained from scratch on a corpus of Finnish news, online discussions, and crawled data.

Download the model here: bert - base - finnish - uncased.zip

Release 0.1

September 30, 2019 We release a beta version of the BERT base cased model trained from scratch on a corpus of Finnish news, online discussions, and crawled data.

Download the model here: bert - base - finnish - cased.zip

Featured Recommended AI Models

Empowering the Future, Your AI Solution Knowledge Base

English 简体中文繁體中文にほんご