Ruadaptqwen2.5 7B Lite Beta

🚀 Russian Adaptation of T-lite-it-1.0 Model

This project focuses on adapting the T-lite-it-1.0 model to the Russian language, enhancing its text - generation capabilities for Russian users.

🚀 Quick Start

You can try out the model in the deployed Space: https://huggingface.co/spaces/RefalMachine/RuadaptQwen2.5

✨ Features

Model Adaptation

This is an adaptation of the T-lite-it-1.0 model to the Russian language. The tokenizer in the model was replaced, followed by continued pretraining on a Russian corpus, and then the Learned Embedding Propagation (LEP) technique was applied.

Improved Generation Speed

Thanks to the new tokenizer (an extended tiktoken cl100k with a unigram tokenizer of 48k tokens), the generation speed* of Russian texts has increased by up to 60% compared to the original T-lite-it-1.0 model.

*Generation speed refers to the number of Russian characters/words per second for the same text sequences.

📦 Installation

No installation steps are provided in the original document.

💻 Usage Examples

No code examples are provided in the original document.

📚 Documentation

Datasets

• IlyaGusev/saiga_scored
• IlyaGusev/saiga_preferences
• dichspace/darulm

Language

• ru

Pipeline Tag

text - generation

License

apache - 2.0

Base Model

• Qwen/Qwen2.5 - 7B
• t - tech/T - lite - it - 1.0

Tokenization

Metrics and Quality Assessment

The model was evaluated on Ru - Arena - General, MERA, llmtf_open.

Results on Ru - Arena - General

Measurements were made using the official leaderboard code (https://github.com/VikhrModels/ru_llm_arena), but with repetition_penalty = 1.1.

Results on Shlepa

Results on MERA

Results on llmtf_open

TODO

🔧 Technical Details

WORK IN PROGRESS!!! The current version is v1.

📄 License

The model is licensed under the apache - 2.0 license.

How to cite:

Tikhomirov M., Chernyshov D. Facilitating Large Language Model Russian Adaptation with Learned Embedding Propagation //Journal of Language and Education. – 2024. – Vol. 10. – No. 4. – Pp. 130 - 145.

Tikhomirov M., Chernyshev D. Impact of Tokenization on LLaMa Russian Adaptation //2023 Ivannikov Ispras Open Conference (ISPRAS). – IEEE, 2023. – Pp. 163 - 168.

⚠️ Important Note

The model's responses do not reflect the opinions of the authors but merely repeat the knowledge obtained from the data at all stages of training (pretraining, tokenizer replacement, instruction - based training, answer - quality calibration). The model was derived from a third - party pre - trained model, and the current authors are not responsible for the pre - training of this model. No additional actions were taken to change the "opinions" embedded in the LLM when creating this version of the model. Use it with caution.