Ruadaptqwen2.5 32B Instruct

🚀 RuadaptQwen2.5

An instruction-tuned version of Qwen2.5-32B adapted for the Russian language, achieving up to 60% faster Russian text generation.

🚀 Quick Start

You can try out the model on the hosted Space: https://huggingface.co/spaces/RefalMachine/RuadaptQwen2.5

✨ Features

• Enhanced Tokenizer: Replaced the original tokenizer and continued pretraining on a Russian corpus.
• Faster Generation: Achieved up to 60% faster Russian text generation compared to the original Qwen-2.5-32B-Instruct model.

📦 Installation

No installation steps were provided in the original document, so this section is skipped.

💻 Usage Examples

No code examples were provided in the original document, so this section is skipped.

📚 Documentation

Model Description

WORK IN PROGRESS!!! The current version is v1.

This is an instruction-tuned version of the Qwen2.5-32B model adapted for the Russian language. The tokenizer was replaced, followed by continued pretraining on a Russian corpus. Then, the Learned Embedding Propagation (LEP) technique was applied (paper will be released soon).

Thanks to the new tokenizer (an extended tiktoken cl100k with a unigram tokenizer of 48k tokens), the generation speed* of Russian texts has increased by up to 60% compared to the original Qwen-2.5-32B-Instruct model.

*Generation speed refers to the number of Russian characters/words per second for the same text sequences.

Tokenization

Metrics and Quality Evaluation

The model was evaluated on Ru-Arena-General, MERA, and llmtf_open.

Results on Ru-Arena-General

Measurements were taken using the official leaderboard code (https://github.com/VikhrModels/ru_llm_arena), but with repetition_penalty = 1.1.

Results on MERA

A custom system prompt was prepared for the MERA submission, which mitigates the shortcomings of the evaluation on coding tasks. For comparison, a submission was also made with the same system prompt for the original model.

Results on llmtf_open

TODO

How to cite:

• Tikhomirov M., Chernyshev D. Facilitating large language model Russian adaptation with Learned Embedding Propagation // 2024 (Preprint: https://arxiv.org/abs/2412.21140)
• Tikhomirov M., Chernyshev D. Impact of Tokenization on LLaMa Russian Adaptation //2023 Ivannikov Ispras Open Conference (ISPRAS). – IEEE, 2023. – P. 163-168.

🔧 Technical Details

No detailed technical implementation details were provided in the original document, so this section is skipped.

📄 License

This project is licensed under the Apache-2.0 license.

⚠️ Important Note

The model's responses do not reflect the opinions of the authors but merely repeat the knowledge obtained from the data at all stages of training (pretraining, tokenizer replacement, instruction training, response quality calibration). The model was derived from a third - party pretrained model, and the current authors are not responsible for the pretraining of this model. No additional actions were taken to change the "opinions" embedded in the LLM when creating this version of the model. Use with caution.