Ruroberta Large Paraphrase V1

🚀 Russian Cross-Encoder for Sentence Semantic Equivalence

This is a cross-encoder model designed to predict the semantic equivalence of two Russian sentences. It can classify text pairs as paraphrases (class 1) or non - paraphrases (class 0). The scores produced by this model can serve as a metric for content preservation in paraphrasing or text style transfer tasks.

🚀 Quick Start

This is a sberbank-ai/ruRoberta-large model fine - tuned on a combination of 3 datasets:

1. RuPAWS: https://github.com/ivkrotova/rupaws_dataset based on Quora and QQP;
2. ru_paraphraser: https://huggingface.co/merionum/ru_paraphraser;
3. Results of the manual check of content preservation for the RUSSE - 2022 text detoxification dataset collection (content_5.tsv).

The task was framed as a binary classification problem: determining whether two sentences have the same meaning (1) or different meanings (0).

✨ Features

• Semantic Equivalence Prediction: Capable of predicting whether two Russian sentences are semantically equivalent.
• Binary Classification: Classifies text pairs into paraphrases and non - paraphrases.
• Content Preservation Metric: The scores can be used to measure content preservation in text transformation tasks.

📦 Installation

No specific installation steps are provided in the original document, so this section is skipped.

💻 Usage Examples

Basic Usage

import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer

model = AutoModelForSequenceClassification.from_pretrained('SkolkovoInstitute/ruRoberta-large-paraphrase-v1')
tokenizer = AutoTokenizer.from_pretrained('SkolkovoInstitute/ruRoberta-large-paraphrase-v1')

def get_similarity(text1, text2):
    """ Predict the probability that two Russian sentences are paraphrases of each other. """
    with torch.inference_mode():
        batch = tokenizer(
            text1, text2, 
            truncation=True, max_length=model.config.max_position_embeddings, return_tensors='pt',
        ).to(model.device)
        proba = torch.softmax(model(**batch).logits, -1)
    return proba[0][1].item()

print(get_similarity('Я тебя люблю', 'Ты мне нравишься'))  # 0.9798
print(get_similarity('Я тебя люблю', 'Я тебя ненавижу'))   # 0.0008

🔧 Technical Details

Training Datasets

The table below shows the training dataset size after duplication (joining text1 + text2 and text2 + text1 pairs):

Training Hyperparameters

The model was trained with the Adam optimizer and the following hyperparameters:

learning_rate = 1e-5
batch_size = 8
gradient_accumulation_steps = 4
n_epochs = 3
max_grad_norm = 1.0

Test Set Performance

After training, the model had the following ROC AUC scores on the test sets: