Roberta Small Bulgarian

🚀 RoBERTa-small-bulgarian

A smaller version of RoBERTa-base-bulgarian with 6 hidden layers and similar performance, suitable for Bulgarian language tasks.

🚀 Quick Start

The RoBERTa model was originally introduced in this paper. This is a smaller version of RoBERTa-base-bulgarian with only 6 hidden layers, but similar performance.

✨ Features

Intended Uses

This model can be used for cloze tasks (masked language modeling) or finetuned on other tasks in Bulgarian.

Limitations and Bias

The training data is unfiltered text from the internet and may contain all sorts of biases.

📦 Installation

No installation steps provided in the original document.

💻 Usage Examples

No code examples provided in the original document.

📚 Documentation

Training Data

This model was trained on the following data:

• bg_dedup from OSCAR
• Newscrawl 1 million sentences 2017 from Leipzig Corpora Collection
• Wikipedia 1 million sentences 2016 from Leipzig Corpora Collection

Training Procedure

The model was pretrained using a masked language-modeling objective with dynamic masking as described here. It was trained for 160k steps. The batch size was limited to 8 due to GPU memory limitations.

🔧 Technical Details

No specific technical details (more than 50 words) provided in the original document.

📄 License

No license information provided in the original document.