Sage Fredt5 Distilled 95m

🚀 sage-fredt5-distilled-95m

The model corrects spelling and punctuation errors and typos in Russian text, bringing all words to the norm of the Russian language.

🚀 Quick Start

The sage-fredt5-distilled-95m model is designed to correct spelling and punctuation errors in Russian text. Here's a basic example of how to use it:

from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

tokenizer = AutoTokenizer.from_pretrained("ai-forever/sage-fredt5-distilled-95m")
model = AutoModelForSeq2SeqLM.from_pretrained("ai-forever/sage-fredt5-distilled-95m")

model.to("cuda")

sentence = "И не чсно прохожим в этот день непогожйи почему я веселый такйо"
inputs = tokenizer(sentence, max_length=None, padding="longest", truncation=False, return_tensors="pt")
outputs = model.generate(**inputs.to(model.device), max_length = inputs["input_ids"].size(1) * 1.5)
print(tokenizer.batch_decode(outputs, skip_special_tokens=True))

# ["И не ясно прохожим в этот день непогожий, почему я весёлый такой?"]

✨ Features

• Spelling and Punctuation Correction: The model can correct spelling and punctuation errors in Russian text, bringing all words to the norm of the Russian language.
• Distilled Model: It is a distilled version of the original model based on the FRED-T5-1.7B architecture, which may offer better performance in terms of speed and resource utilization.
• Extensive Training Data: The model was trained on an extensive dataset with “artificial” errors, assembled from the Russian-language Wikipedia and transcripts of Russian-language videos, with typos and spelling errors automatically introduced using the SAGE library.

📦 Installation

To use the sage-fredt5-distilled-95m model, you need to install the transformers library. You can install it using pip:

pip install transformers

💻 Usage Examples

Basic Usage

from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

tokenizer = AutoTokenizer.from_pretrained("ai-forever/sage-fredt5-distilled-95m")
model = AutoModelForSeq2SeqLM.from_pretrained("ai-forever/sage-fredt5-distilled-95m")

model.to("cuda")

sentence = "И не чсно прохожим в этот день непогожйи почему я веселый такйо"
inputs = tokenizer(sentence, max_length=None, padding="longest", truncation=False, return_tensors="pt")
outputs = model.generate(**inputs.to(model.device), max_length = inputs["input_ids"].size(1) * 1.5)
print(tokenizer.batch_decode(outputs, skip_special_tokens=True))

# ["И не ясно прохожим в этот день непогожий, почему я весёлый такой?"]

Advanced Usage

You can use the model in more complex scenarios, such as processing multiple sentences or integrating it into a larger application. Here's an example of processing multiple sentences:

from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

tokenizer = AutoTokenizer.from_pretrained("ai-forever/sage-fredt5-distilled-95m")
model = AutoModelForSeq2SeqLM.from_pretrained("ai-forever/sage-fredt5-distilled-95m")

model.to("cuda")

sentences = [
    "И не чсно прохожим в этот день непогожйи почему я веселый такйо",
    "Каждй день воттак делой, и спена балеть нибудет. А вотак каждый день ниделай"
]

for sentence in sentences:
    inputs = tokenizer(sentence, max_length=None, padding="longest", truncation=False, return_tensors="pt")
    outputs = model.generate(**inputs.to(model.device), max_length = inputs["input_ids"].size(1) * 1.5)
    print(tokenizer.batch_decode(outputs, skip_special_tokens=True))

📚 Documentation

Public References

• SAGE library announcement, DataFest 2023
• Paper about synthetic error generation methods, Dialogue 2023
• SAGE EACL 2024 paper

Examples

Metrics

Quality

Below are automatic metrics for determining the correctness of the spell checkers. We compare our solution with both open automatic spell checkers and the ChatGPT family of models on all four available datasets:

• RUSpellRU: texts collected from (LiveJournal), with manually corrected typos and errors;
• MultidomainGold: examples from 7 text sources, including the open web, news, social media, reviews, subtitles, policy documents and literary works;
• MedSpellChecker: texts with errors from medical anamnesis;
• GitHubTypoCorpusRu: spelling errors and typos in commits from GitHub;

RUSpellRU

MultidomainGold

MedSpellChecker

GitHubTypoCorpusRu

🔧 Technical Details

• Model Type: Distilled Seq2Seq model based on the FRED-T5 architecture.
• Training Data: An extensive dataset with “artificial” errors, assembled from the Russian-language Wikipedia and transcripts of Russian-language videos, with typos and spelling errors automatically introduced using the SAGE library.
• Framework: PyTorch

Information Table

📄 License

The FRED-T5-1.7B model, on the basis of which our solution is made, and its source code are supplied under the MIT license. Our solution also comes with the MIT license.

📦 Resources

• SAGE library, GitHub
• sage-fredt5-large, HuggingFace
• sage-fredt5-distilled-95m, HuggingFace
• sage-m2m100-1.2B, HuggingFace
• sage-mt5-large, HuggingFace

📞 Contacts

nikita.martynov.98@list.ru

⚠️ Important Note

• Complex formatting may cause some trouble in output generation.

💡 Usage Tip

• Make sure to have a GPU available for better performance when using the model, especially for processing large amounts of text. You can move the model to the GPU using the model.to("cuda") line in the code examples.