Robbertje-1-gb-merged Open-source Dutch Model - Multiple Scale Versions Suitable for Multiple Natural Language Processing Tasks

Robbertje 1 Gb Merged

Developed by DTAI-KULeuven

RobBERTje is a series of Dutch language models distilled from RobBERT, offering versions of different scales and training configurations suitable for various natural language processing tasks.

Large Language Model

Transformers

OtherOpen Source License:MIT #Dutch distilled model #Lightweight BERT #Multi-task adaptation

Downloads 39

Release Time : 3/2/2022

Model Overview

RobBERTje is a collection of distilled models based on RobBERT, specifically optimized for Dutch, providing versions with varying parameter sizes to accommodate different computational needs.

Model Features

Multiple version options

Offers 4 model versions with different training configurations, including non-shuffled, shuffled, merged, and compact BORT versions

Efficient distillation

Learns from the RobBERT teacher model through knowledge distillation, significantly reducing parameters while maintaining performance

Comprehensive evaluation

Thoroughly evaluated on multiple Dutch NLP tasks (DBRD, DIE-DAT, NER, POS, SICK-NL)

Model Capabilities

Dutch text understanding

Masked language modeling

Text classification

Named entity recognition

Part-of-speech tagging

Natural language inference

Use Cases

Academic research

Dutch NLP research

Used as a benchmark model and research tool for Dutch natural language processing tasks

Achieves performance close to the teacher model RobBERT on multiple tasks

Industrial applications

Dutch text processing

Suitable for commercial applications requiring Dutch language understanding, such as customer service bots and content moderation

Provides efficient inference performance, ideal for resource-constrained environments

🚀 RobBERTje

RobBERTje is a collection of distilled Dutch BERT-based models, offering multiple models with different sizes and training settings for various use-cases.

RobBERTje: A collection of distilled Dutch BERT-based models

🚀 Quick Start

RobBERTje provides a variety of distilled models based on RobBERT. You can select different models according to your specific needs. We are constantly working on releasing models with better performance, so stay tuned to the repository for updates.

✨ Features

Multiple models with different sizes and training settings.
Continuously updated with better-performing models.

📚 Documentation

News

February 21, 2022: Our paper about RobBERTje has been published in volume 11 of CLIN journal!
July 2, 2021: Publicly released 4 RobBERTje models.
May 12, 2021: RobBERTje was accepted at CLIN31 for an oral presentation!

The models

Model	Description	Parameters	Training size	Huggingface id
Non-shuffled	Trained on the non-shuffled variant of the oscar corpus, without any operations to preserve this order during training and distillation.	74 M	1 GB	DTAI-KULeuven/robbertje-1-gb-non-shuffled
Shuffled	Trained on the publicly available and shuffled OSCAR corpus.	74 M	1 GB	DTAI-KULeuven/robbertje-1-gb-shuffled
Merged (p=0.5)	Same as the non-shuffled variant, but sequential sentences of the same document are merged with a probability of 50%.	74 M	1 GB	this model
BORT	A smaller version with 8 attention heads instead of 12 and 4 layers instead of 6 (and 12 for RobBERT).	46 M	1 GB	DTAI-KULeuven/robbertje-1-gb-bort

Results

Intrinsic results

We calculated the pseudo perplexity (PPPL) from cite, which is a built-in metric in our distillation library. This metric gives an indication of how well the model captures the input distribution.

Model	PPPL
RobBERT (teacher)	7.76
Non-shuffled	12.95
Shuffled	18.74
Merged (p=0.5)	17.10
BORT	26.44

Extrinsic results

We also evaluated our models on several downstream tasks, just like the teacher model RobBERT. Since that evaluation, a Dutch NLI task named SICK-NL was also released and we evaluated our models with it as well.

Model	DBRD	DIE-DAT	NER	POS	SICK-NL
RobBERT (teacher)	94.4	99.2	89.1	96.4	84.2
Non-shuffled	90.2	98.4	82.9	95.5	83.4
Shuffled	92.5	98.2	82.7	95.6	83.4
Merged (p=0.5)	92.9	96.5	81.8	95.2	82.8
BORT	89.6	92.2	79.7	94.3	81.0

📄 License

This project is licensed under the MIT license.

Featured Recommended AI Models

Empowering the Future, Your AI Solution Knowledge Base

English 简体中文繁體中文にほんご