Ro Bart 512

🚀 Pretrained BART Base Model

This is a BART base model (140M parameters) that has been pretrained from scratch. It offers potential for various natural language processing tasks, especially in the Romanian language context.

🚀 Quick Start

This is a BART base model (140M parameters) that has been pretrained from scratch.

✨ Features

• Pretrained from Scratch: Built from the ground - up to capture the nuances of the Romanian language.
• Large - Scale Training: Trained on a clean 50GB Romanian text corpus.
• Defined Sequence Length: Trained with a maximum sequence length of 512.

📦 Installation

No installation steps are provided in the original document.

💻 Usage Examples

No code examples are provided in the original document.

📚 Documentation

Training was performed on a clean 50GB Romanian text corpus for 3M steps with these scripts. The model was trained with a maximum sequence length of 512.

🔧 Technical Details

The model is a BART base model with 140M parameters. It was trained on a large Romanian text corpus for 3M steps using specific scripts. The maximum sequence length during training was set to 512.

📄 License

No license information is provided in the original document.

⚠️ Important Note

This model was pretrained on the text corruption task, meaning this model is not usable in any downstream task without finetuning first!