Ru Rope T5 Small Instruct

🚀 ru-rope-t5-small-instruct

The Russian Rotary Position Embedding T5 model of small version after instruct tuning, designed for various NLP tasks with a mix of Russian and English.

🚀 Quick Start

This README provides detailed information about the ru-rope-t5-small-instruct model, including its details, uses, training information, and more.

✨ Features

• Trained in a Russian corpus with a mix of English using the Mixture-Of-Denoisers pre - training method by UL2 on 1024 length sequences.
• Training using Flash Attention 2 is available because of the replacement of bias with rotary encoding.

📚 Documentation

Model Details

The model was trained in a Russian corpus with a mix of English using the Mixture-Of-Denoisers pre - training method by UL2 on 1024 length sequences. Training using Flash Attention 2 is available because of the replacement of bias with rotary encoding.

Uses

Finetuning for downstream tasks

Bias, Risks, and Limitations

⚠️ Important Note

Despite the instructional tuning, it is not recommended to use in zero - shot mode due to the small size

Training Details

Training Data

A corpus of Russian texts from Vikhr filtered by FRED-T5-1.7B perplexy. Instructions are translated English set

Training Procedure

Using AdamWScale instead of Adafactor for stable learning without loss explosions

Metrics

Model Card Contact

You can contact the model card maintainer via @TheMelmoth

📄 License

This project is licensed under the Apache-2.0 license.