đ RobBERTje
RobBERTje is a collection of distilled Dutch BERT - based models, offering multiple options with different sizes and training settings for various use - cases.
đ Quick Start
RobBERTje provides a range of distilled models based on RobBERT. You can choose from multiple models with different sizes and training settings according to your specific use - case. We are constantly working on releasing models with better performance. Keep an eye on the repository for updates.
đ Documentation
News
- February 21, 2022: Our paper about RobBERTje has been published in volume 11 of CLIN journal!
- July 2, 2021: Publicly released 4 RobBERTje models.
- May 12, 2021: RobBERTje was accepted at CLIN31 for an oral presentation!
The models
| Property |
Details |
| Model Type |
Non - shuffled, Shuffled, Merged (p = 0.5), BORT |
| Description |
|
- Non - shuffled: Trained on the non - shuffled variant of the oscar corpus, without any operations to preserve this order during training and distillation.
- Shuffled: Trained on the publicly available and shuffled OSCAR corpus.
- Merged (p = 0.5): Same as the non - shuffled variant, but sequential sentences of the same document are merged with a probability of 50%.
- BORT: A smaller version with 8 attention heads instead of 12 and 4 layers instead of 6 (and 12 for RobBERT). |
|
| Parameters |
Non - shuffled: 74 M; Shuffled: 74 M; Merged (p = 0.5): 74 M; BORT: 46 M |
| Training size |
All models: 1 GB |
| Huggingface id |
|
- Non - shuffled: [DTAI - KULeuven/robbertje - 1 - gb - non - shuffled](https://huggingface.co/DTAI - KULeuven/robbertje - 1 - gb - non - shuffled)
- Shuffled: this model
- Merged (p = 0.5): [DTAI - KULeuven/robbertje - 1 - gb - merged](https://huggingface.co/DTAI - KULeuven/robbertje - 1 - gb - merged)
- BORT: [DTAI - KULeuven/robbertje - 1 - gb - bort](https://huggingface.co/DTAI - KULeuven/robbertje - 1 - gb - bort) |
|
Results
Intrinsic results
We calculated the pseudo perplexity (PPPL) from cite, which is a built - in metric in our distillation library. This metric gives an indication of how well the model captures the input distribution.
| Model |
PPPL |
| RobBERT (teacher) |
7.76 |
| Non - shuffled |
12.95 |
| Shuffled |
18.74 |
| Merged (p = 0.5) |
17.10 |
| BORT |
26.44 |
Extrinsic results
We also evaluated our models on several downstream tasks, just like the teacher model RobBERT. Since that evaluation, a Dutch NLI task named SICK - NL was also released and we evaluated our models with it as well.
| Model |
DBRD |
DIE - DAT |
NER |
POS |
SICK - NL |
| RobBERT (teacher) |
94.4 |
99.2 |
89.1 |
96.4 |
84.2 |
| Non - shuffled |
90.2 |
98.4 |
82.9 |
95.5 |
83.4 |
| Shuffled |
92.5 |
98.2 |
82.7 |
95.6 |
83.4 |
| Merged (p = 0.5) |
92.9 |
96.5 |
81.8 |
95.2 |
82.8 |
| BORT |
89.6 |
92.2 |
79.7 |
94.3 |
81.0 |
đ License
This project is licensed under the MIT license.
%20--%3e%3cdefs%3e%3cstyle%3e%20.st0%20{%20fill:%20%23061b40;%20}%20.st1%20{%20fill:%20%23306af1;%20}%20.st2%20{%20fill:%20%235ce5cf;%20}%20%3c/style%3e%3c/defs%3e%3cg%3e%3cpath%20class='st0'%20d='M55,10.5h9v3h-9v9h12V7.5h-12v3ZM64,19.5h-6v-3h6v3Z'/%3e%3cpolygon%20class='st0'%20points='69%2016.5%2078%2016.5%2078%2019.5%2069%2019.5%2069%2022.5%2081%2022.5%2081%2013.5%2072%2013.5%2072%2010.5%2081%2010.5%2081%207.5%2069%207.5%2069%2016.5'/%3e%3cpolygon%20class='st0'%20points='95%2010.5%2095%207.5%2083%207.5%2083%2022.5%2095%2022.5%2095%2019.5%2086%2019.5%2086%2016.5%2095%2016.5%2095%2013.5%2086%2013.5%2086%2010.5%2095%2010.5'/%3e%3cpath%20class='st0'%20d='M40,1.5v21h11.6l1.4-1.4v-7.6h0c0,0-1.4-1.5-1.4-1.5l1.4-1.4V2.9l-1.4-1.4h-11.6ZM50,19.5h-7v-6h7v6ZM50,10.5h-7v-6h7v6Z'/%3e%3c/g%3e%3cpath%20class='st1'%20d='M23.1,24L14.7,4.8l-4.9,11.2h3.8l-1.8,4H3.3L12.1,0H2C.9,0,0,.9,0,2v20c0,1.1.9,2,2,2h21.1Z'/%3e%3cpath%20class='st2'%20d='M34,0h-16.8l10.6,24h6.2c1.1,0,2-.9,2-2V2C36,.9,35.1,0,34,0ZM32.5,20h-4V4h4v16Z'/%3e%3c/svg%3e)
