Russian Text Normalizer

🚀 Russian Text Normalization

This is a finetuned version of FRED-T5 large 820M, aiming to solve the problem of Russian text normalization and providing a reliable text normalization solution.

The code repository can be found here. This model is trained on sentences from ficbook, librusec, and pikabu. The inverse text normalization is done using a modified version of the word_to_number_ru package. Additionally, data from the Kaggle Text Normalization Challenge is used to enable the normalization of Latin characters. Although the performance on numbers has decreased, it can successfully process Latin characters. Use git checkout #8c2476b to choose the previous number - only version.

🚀 Quick Start

✨ Features

• Finetuned from FRED-T5 large 820M.
• Trained on multiple datasets including ficbook, librusec, and pikabu.
• Can handle both numbers and Latin characters in Russian text normalization.

📦 Installation

No specific installation steps are provided in the original document, so this section is skipped.

💻 Usage Examples

Basic Usage

Put numbers and Latin words inside square brackets and follow with T5 tokens <extra_id_0>, <extra_id_1>, etc. It is recommended to split numbers into groups of 3 digits and only put numbers and Latin inside the [ ] brackets.

import torch
from transformers import GPT2Tokenizer, T5ForConditionalGeneration 

device='cuda'
tokenizer = GPT2Tokenizer.from_pretrained('saarus72/russian_text_normalizer', eos_token='</s>')
model = T5ForConditionalGeneration.from_pretrained('saarus72/russian_text_normalizer').to(device)
lm_text = '<SC1>Было у отца [3]<extra_id_0> сына, но не было даже [2- 3]<extra_id_1> пиджаков с блёстками за [142 990]<extra_id_2> руб.'
input_ids = torch.tensor([tokenizer.encode(lm_text)]).to(device)
outputs = model.generate(input_ids, eos_token_id=tokenizer.eos_token_id, early_stopping=True)
print(tokenizer.decode(outputs[0][1:]))
# <extra_id_0>  три\n <extra_id_1>  двух- трёх\n <extra_id_2>  сто сорок две тысячи девятьсот девяносто\n</s>

More Examples

📄 License

This project is licensed under the Apache-2.0 license.