Ukr Roberta Base

🚀 ukr-roberta-base

This is a pre - trained Ukrainian Roberta model, which uses a variety of corpora for pre - training and follows a specific model architecture.

🚀 Quick Start

The detailed quick - start guide is not provided in the original document.

📚 Documentation

🔍 Pre - training corpora

Below is the list of corpora used along with the output of the wc command (counting lines, words, and characters). These corpora were concatenated and tokenized with the HuggingFace Roberta Tokenizer.

🔧 Pre - training details

• Ukrainian Roberta was trained with code provided in [HuggingFace tutorial](https://huggingface.co/blog/how - to - train).
• Currently released model follows the roberta - base - cased model architecture (12 - layer, 768 - hidden, 12 - heads, 125M parameters).
• The model was trained on 4xV100 (85 hours).
• The training configuration can be found in the [original repository](https://github.com/youscan/language - models).

📄 Author

Vitalii Radchenko - contact me on Twitter @vitaliradchenko