robbertje-1-gb-non-shuffled Open-source Dutch Model - Multiple Versions to Suit Different Application Needs

Robbertje 1 Gb Non Shuffled

Developed by DTAI-KULeuven

RobBERTje is a series of distilled Dutch language models based on RobBERT, offering various versions with different sizes and training configurations.

Large Language Model

Transformers

OtherOpen Source License:MIT #Dutch distilled model #Lightweight BERT #Multi-task adaptation

Downloads 43

Release Time : 3/2/2022

Model Overview

RobBERTje is a series of distilled Dutch language models based on RobBERT, suitable for various natural language processing tasks.

Model Features

Multiple training versions

Provides four different training configurations: non-scrambled, scrambled, merged, and BORT-lite versions

Efficient distillation

Learns from the RobBERT teacher model through knowledge distillation, reducing model size while maintaining performance

Dutch language optimization

Specifically trained and optimized for Dutch, supporting Flemish variants

Model Capabilities

Text classification

Named entity recognition

Part-of-speech tagging

Natural language inference

Sentiment analysis

Use Cases

Text analysis

Sentiment analysis

Analyze sentiment tendencies in Dutch text

Achieved 90.2% accuracy on DBRD dataset (non-scrambled version)

Named entity recognition

Identify named entities in Dutch text

Achieved 82.9% accuracy on CONLL2002 dataset (non-scrambled version)

Language understanding

Natural language inference

Determine logical relationships between Dutch sentences

Achieved 83.4% accuracy on SICK-NL dataset (non-scrambled version)

🚀 RobBERTje: A Collection of Distilled Dutch BERT-based Models

RobBERTje is a collection of distilled models based on RobBERT. It offers multiple models with different sizes and training settings, allowing users to choose the most suitable one for their specific use - cases. The team is also constantly working on releasing better - performing models.

RobBERTje: A collection of distilled Dutch BERT-based models

🚀 Quick Start

This section provides an overview of RobBERTje and its capabilities. For more detailed usage, please refer to the relevant sections below.

✨ Features

A collection of distilled models based on RobBERT.
Multiple models with different sizes and training settings to meet various use - cases.
Continuous efforts to release better - performing models.

📚 Documentation

🔍 About RobBERTje

RobBERTje is a collection of distilled models based on RobBERT. There are multiple models with different sizes and different training settings, which you can choose for your use - case.

We are also continuously working on releasing better - performing models, so watch the repository for updates.

📰 News

February 21, 2022: Our paper about RobBERTje has been published in volume 11 of CLIN journal!
July 2, 2021: Publicly released 4 RobBERTje models.
May 12, 2021: RobBERTje was accepted at CLIN31 for an oral presentation!

🧠 The models

Model	Description	Parameters	Training size	Huggingface id
Non - shuffled	Trained on the non - shuffled variant of the oscar corpus, without any operations to preserve this order during training and distillation.	74 M	1 GB	this model
Shuffled	Trained on the publicly available and shuffled OSCAR corpus.	74 M	1 GB	[DTAI - KULeuven/robbertje - 1 - gb - shuffled](https://huggingface.co/DTAI - KULeuven/robbertje - 1 - gb - shuffled)
Merged (p = 0.5)	Same as the non - shuffled variant, but sequential sentences of the same document are merged with a probability of 50%.	74 M	1 GB	[DTAI - KULeuven/robbertje - 1 - gb - merged](https://huggingface.co/DTAI - KULeuven/robbertje - 1 - gb - merged)
BORT	A smaller version with 8 attention heads instead of 12 and 4 layers instead of 6 (and 12 for RobBERT).	46 M	1 GB	[DTAI - KULeuven/robbertje - 1 - gb - bort](https://huggingface.co/DTAI - KULeuven/robbertje - 1 - gb - bort)

📊 Results

🔍 Intrinsic results

We calculated the pseudo perplexity (PPPL) from cite, which is a built - in metric in our distillation library. This metric gives an indication of how well the model captures the input distribution.

Model	PPPL
RobBERT (teacher)	7.76
Non - shuffled	12.95
Shuffled	18.74
Merged (p = 0.5)	17.10
BORT	26.44

🔎 Extrinsic results

We also evaluated our models on several downstream tasks, just like the teacher model RobBERT. Since that evaluation, a Dutch NLI task named SICK - NL was also released and we evaluated our models with it as well.

Model	DBRD	DIE - DAT	NER	POS	SICK - NL
RobBERT (teacher)	94.4	99.2	89.1	96.4	84.2
Non - shuffled	90.2	98.4	82.9	95.5	83.4
Shuffled	92.5	98.2	82.7	95.6	83.4
Merged (p = 0.5)	92.9	96.5	81.8	95.2	82.8
BORT	89.6	92.2	79.7	94.3	81.0

📄 License

This project is licensed under the MIT license.

Featured Recommended AI Models

Empowering the Future, Your AI Solution Knowledge Base

English 简体中文繁體中文にほんご