Roberta Med Small 1M 1

🚀 RoBERTa Pretrained on Smaller Datasets

This project focuses on pretraining RoBERTa on smaller datasets (1M, 10M, 100M, 1B tokens). We release 3 models with the lowest perplexities for each pretraining data size out of 25 runs (or 10 in the case of 1B tokens). The pretraining data mimics that of BERT, combining English Wikipedia and a reproduction of BookCorpus using texts from smashwords in an approximate ratio of 3:1.

✨ Features

• Pretraining RoBERTa on various smaller datasets.
• Releasing models with the lowest perplexities for each dataset size.
• Reproducing the pretraining data similar to BERT.

📚 Documentation

Hyperparameters and Validation Perplexity

The hyperparameters and validation perplexities corresponding to each model are presented in the following table:

The hyperparameters corresponding to the model sizes mentioned above are as follows:

(AH = number of attention heads; HS = hidden size; FFN = feedforward network dimension; P = number of parameters.)

For other hyperparameters, the following values are selected:

• Peak Learning rate: 5e-4
• Warmup Steps: 6% of max steps
• Dropout: 0.1