Rugpt3large Based On Gpt2

🚀 rugpt3large_based_on_gpt2

This project presents a pre - trained Transformer language model for Russian. It details the model's architecture, training process, and evaluation results.

🚀 Quick Start

The design of the model architecture, the pretraining process, and the evaluation are all documented in our preprint: A Family of Pretrained Transformer Language Models for Russian.

The model was trained by the SberDevices team using the transformers library. It was trained for 3 epochs on 80B tokens with a sequence length of 1024. Subsequently, the model was fine - tuned for 1 epoch with a sequence length of 2048.

The total training time was approximately 14 days on 128 GPUs for a context length of 1024 and a few days on 16 GPUs for a context length of 2048. The final perplexity on the test set is 13.6.

👥 Authors

• NLP core team RnD Telegram channel:
  • Dmitry Zmitrovich

📚 Citation

@misc{zmitrovich2023family,
      title={A Family of Pretrained Transformer Language Models for Russian}, 
      author={Dmitry Zmitrovich and Alexander Abramov and Andrey Kalmykov and Maria Tikhonova and Ekaterina Taktasheva and Danil Astafurov and Mark Baushenko and Artem Snegirev and Tatiana Shavrina and Sergey Markov and Vladislav Mikhailov and Alena Fenogenova},
      year={2023},
      eprint={2309.10931},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}