Bertin Base Gaussian

🚀 RoBERTa-base Model in Spanish

This is a RoBERTa-base model trained from scratch in Spanish, which can effectively solve the fill - mask task in Spanish text, offering high - quality language processing capabilities.

🚀 Quick Start

This is a RoBERTa-base model trained from scratch in Spanish. It can be used for fill - mask tasks. For more detailed usage, please refer to the official card.

✨ Features

• Trained from Scratch: Built from the ground up in Spanish, ensuring better adaptation to the Spanish language.
• Quality Sampling: The training dataset is carefully sampled to improve the quality of the model.

📦 Installation

No specific installation steps are provided in the original document, so this section is skipped.

💻 Usage Examples

No code examples are provided in the original document, so this section is skipped.

📚 Documentation

The training dataset is mc4 subsampling documents to a total of about 50 million examples. Sampling is biased towards average perplexity values (using a Gaussian function), discarding more often documents with very large values (poor quality) of very small values (short, repetitive texts).

This model has been trained for 250,000 steps.

Please see our main card for more information.

This is part of the Flax/Jax Community Week, organised by HuggingFace and TPU usage sponsored by Google.

🔧 Technical Details

The model samples the training dataset mc4 to about 50 million examples. It uses a Gaussian function to bias towards average perplexity values during sampling, which helps in filtering out low - quality and short, repetitive documents. The model is trained for 250,000 steps.

📄 License

The model is licensed under CC - BY - 4.0.

Team members

• Eduardo González (edugp)
• Javier de la Rosa (versae)
• Manu Romero (mrm8488)
• María Grandury (mariagrandury)
• Pablo González de Prado (Pablogps)
• Paulo Villegas (paulo)