Ruadaptqwen2.5 32B Pro Beta

🚀 Russian Adapted Qwen2.5 Model

This project focuses on adapting the T-pro-it-1.0 model to the Russian language, enhancing the generation speed of Russian texts through techniques such as tokenizer replacement and continued pretraining.

🚀 Quick Start

You can try out the model in the deployed Space: https://huggingface.co/spaces/RefalMachine/RuadaptQwen2.5

✨ Features

Model Adaptation

• Adapted the T-pro-it-1.0 model to the Russian language.
• Replaced the tokenizer and performed continued pretraining on a Russian corpus.
• Applied the Learned Embedding Propagation (LEP) technique.

Improved Generation Speed

• Thanks to the new tokenizer (an extended tiktoken cl100k with a unigram tokenizer of 48k tokens), the generation speed* of Russian texts has increased by up to 60% compared to the original T-pro-it-1.0 model.
• *Generation speed refers to the number of Russian characters/words per second for the same text sequences.

📚 Documentation

Datasets

• IlyaGusev/saiga_scored
• IlyaGusev/saiga_preferences
• dichspace/darulm

Language

• ru

Pipeline Tag

• text-generation

License

• apache-2.0

Base Model

• Qwen/Qwen2.5-32B
• t-tech/T-pro-it-1.0

Tokenization

Metrics and Quality Evaluation

The model was evaluated on Ru-Arena-General, MERA, and llmtf_open.

Results on Ru-Arena-General

Measurements were taken using the official leaderboard code (https://github.com/VikhrModels/ru_llm_arena), but with repetition_penalty = 1.1.

Results on MERA

TODO

Results on llmtf_open

TODO

How to cite:

Tikhomirov M., Chernyshov D. Facilitating Large Language Model Russian Adaptation with Learned Embedding Propagation //Journal of Language and Education. – 2024. – Vol. 10. – No. 4. – Pp. 130-145. Tikhomirov M., Chernyshev D. Impact of Tokenization on LLaMa Russian Adaptation //2023 Ivannikov Ispras Open Conference (ISPRAS). – IEEE, 2023. – Pp. 163-168.

📄 License

This project is licensed under the Apache-2.0 license.

⚠️ Important Note

The model's responses do not reflect the opinions of the authors, but merely repeat the knowledge obtained from the data at all stages of training (pretraining, tokenizer replacement, instruction training, and response quality calibration). The model was derived from a third - party pretrained model, and the current authors are not responsible for the pretraining of this model. No additional actions were taken to change the "opinions" embedded in the LLM when creating this version of the model. Use with caution.