đ Continued Pre - training of MedRoBERTa.nl
This project focuses on the continued, off - premise pre - training of MedRoBERTa.nl using approximately 50GB of open Dutch and translated English corpora, followed by on - premise pre - training on a combination of 5GB of Electronic Health records and 2GB of the public set.
đ Quick Start
The project involves continued pre - training of the MedRoBERTa.nl model. First, off - premise pre - training is conducted with about 50GB of open Dutch and translated English corpora. Then, on - premise pre - training is carried out on a mix of 5GB of Electronic Health records and 2GB of the public set.
⨠Features
- Continued pre - training of the medical model MedRoBERTa.nl.
- Utilization of a large amount of Dutch and English corpora from diverse medical sources.
- A combination of off - premise and on - premise pre - training strategies.
đĻ Installation
No specific installation steps are provided in the original document, so this section is skipped.
đģ Usage Examples
No code examples are provided in the original document, so this section is skipped.
đ Documentation
Data statistics
Sources:
- Dutch: medical guidelines (FMS, NHG)
- Dutch: NtvG papers
- Dutch: Cardiovascular Electronic Health Records
- English: Pubmed abstracts
- English: PMC abstracts translated using DeepL
- English: Apollo guidelines, papers and books
- English: Meditron guidelines
- English: MIMIC3
- English: MIMIC CXR
- English: MIMIC4
All data that is translated (if not with DeepL) is done using a combination of GeminiFlash 1.5/2.0/GPT4o mini, MariaNMT, NLLB200.
- Number of tokens: 20B
- Number of documents: 32M
Training
- Effective batch size: 5120
- Learning rate: 2e - 4
- Weight decay: 1e - 3
- Learning schedule: linear, with 5,000 warmup steps
- Num epochs: ~3 (off - premise) followed by 3 (on - premise)
Perplexity:
- Train perplexity: 2.4
- Validation perplexity: 3.3
Acknowledgement
This work was done in collaboration with the Amsterdam UMC, within the context of the DataTools4Heart project.
The project team was fortunate to use the Google TPU research cloud for model training.
đ§ Technical Details
The project conducts continued pre - training of the MedRoBERTa.nl model. It uses a large amount of Dutch and English medical corpora from various sources. The data translation process involves multiple translation tools. The training process has specific hyperparameters such as batch size, learning rate, weight decay, and a linear learning schedule with warm - up steps. The model is trained in two phases: off - premise and on - premise, and the perplexity metrics are used to evaluate the training and validation performance.
đ License
The project is licensed under the GPL - 3.0 license.
| Property |
Details |
| Model Type |
Continued pre - trained MedRoBERTa.nl |
| Training Data |
Dutch and English medical corpora from multiple sources, including medical guidelines, papers, and Electronic Health Records |
| License |
GPL - 3.0 |
| Library Name |
transformers |
| Metrics |
Perplexity |
| Base Model |
CLTL/MedRoBERTa.nl |
| Tags |
medical, healthcare |
%20--%3e%3cdefs%3e%3cstyle%3e%20.st0%20{%20fill:%20%23061b40;%20}%20.st1%20{%20fill:%20%23306af1;%20}%20.st2%20{%20fill:%20%235ce5cf;%20}%20%3c/style%3e%3c/defs%3e%3cg%3e%3cpath%20class='st0'%20d='M55,10.5h9v3h-9v9h12V7.5h-12v3ZM64,19.5h-6v-3h6v3Z'/%3e%3cpolygon%20class='st0'%20points='69%2016.5%2078%2016.5%2078%2019.5%2069%2019.5%2069%2022.5%2081%2022.5%2081%2013.5%2072%2013.5%2072%2010.5%2081%2010.5%2081%207.5%2069%207.5%2069%2016.5'/%3e%3cpolygon%20class='st0'%20points='95%2010.5%2095%207.5%2083%207.5%2083%2022.5%2095%2022.5%2095%2019.5%2086%2019.5%2086%2016.5%2095%2016.5%2095%2013.5%2086%2013.5%2086%2010.5%2095%2010.5'/%3e%3cpath%20class='st0'%20d='M40,1.5v21h11.6l1.4-1.4v-7.6h0c0,0-1.4-1.5-1.4-1.5l1.4-1.4V2.9l-1.4-1.4h-11.6ZM50,19.5h-7v-6h7v6ZM50,10.5h-7v-6h7v6Z'/%3e%3c/g%3e%3cpath%20class='st1'%20d='M23.1,24L14.7,4.8l-4.9,11.2h3.8l-1.8,4H3.3L12.1,0H2C.9,0,0,.9,0,2v20c0,1.1.9,2,2,2h21.1Z'/%3e%3cpath%20class='st2'%20d='M34,0h-16.8l10.6,24h6.2c1.1,0,2-.9,2-2V2C36,.9,35.1,0,34,0ZM32.5,20h-4V4h4v16Z'/%3e%3c/svg%3e)
