rubert-base-cased-dp-paraphrase-detection Open Source Model - Detect Whether Russian Texts Are in a Paraphrasing Relationship

Rubert Base Cased Dp Paraphrase Detection

Developed by cointegrated

This is a paraphrase detector developed based on DeepPavlov, migrated to the Transformers format, used to detect whether Russian texts are paraphrases.

Text Classification

Transformers

Other#Russian Paraphrase Detection #Text Similarity #BERT Architecture

Downloads 39

Release Time : 3/2/2022

Model Overview

This model can classify Russian text pairs as paraphrases (class 1) or non-paraphrases (class 0), primarily used for text similarity comparison and paraphrase detection tasks.

Model Features

Russian Paraphrase Detection

A paraphrase detection model specifically designed for Russian texts, capable of accurately identifying semantically similar text pairs.

BERT Architecture

Utilizes BERT pre-trained model as the base architecture, offering powerful semantic understanding capabilities.

Transformers Compatibility

Converted to Hugging Face Transformers format for easy integration into modern NLP workflows.

Model Capabilities

Text similarity comparison

Paraphrase detection

Russian text classification

Use Cases

Text Processing

Paraphrase Identification

Identify texts with different expressions but the same meaning

Can accurately distinguish between paraphrase and non-paraphrase text pairs

Content Deduplication

Detect different expressions of duplicate content

Helps reduce redundant information

Education

Homework Checking

Detect paraphrased content in student assignments

Identify potential plagiarism or simple rewrites

🚀 Paraphrase Detector Ported to Transformers

This is a Transformers-formatted version of the paraphrase detector by DeepPavlov, which can classify texts as paraphrases or non - paraphrases.

🚀 Quick Start

This is a ported version of the paraphrase detector by DeepPavlov. You can find more details in the documentation. All the credit for the original work goes to the authors of DeepPavlov.

The model was trained on the dataset from http://paraphraser.ru/. It classifies texts into two categories: paraphrases (class 1) and non - paraphrases (class 0).

💻 Usage Examples

Basic Usage

import torch
from transformers import AutoModelForSequenceClassification, BertTokenizer
model_name = 'cointegrated/rubert-base-cased-dp-paraphrase-detection'
model = AutoModelForSequenceClassification.from_pretrained(model_name).cuda()
tokenizer = BertTokenizer.from_pretrained(model_name)

def compare_texts(text1, text2):
    batch = tokenizer(text1, text2, return_tensors='pt').to(model.device)
    with torch.inference_mode():
        proba = torch.softmax(model(**batch).logits, -1).cpu().numpy()
    return proba[0] # p(non-paraphrase), p(paraphrase)

print(compare_texts('Сегодня на улице хорошая погода', 'Сегодня на улице отвратительная погода'))
# [0.7056226 0.2943774]
print(compare_texts('Сегодня на улице хорошая погода', 'Отличная погодка сегодня выдалась'))
# [0.16524374 0.8347562 ]

Advanced Usage

P.S. In the DeepPavlov repository, the tokenizer uses max_seq_length = 64. However, this model uses model_max_length = 512. Therefore, the results on long texts may be inadequate.

📚 Documentation

Model Information

Property	Details
Model Type	Paraphrase Detector
Training Data	Dataset from http://paraphraser.ru/
Tags	sentence - similarity, text - classification

Important Note

⚠️ Important Note

The tokenizer in the DeepPavlov repository uses max_seq_length = 64, while this model uses model_max_length = 512. So, the results on long texts may be unreliable.

Featured Recommended AI Models

Empowering the Future, Your AI Solution Knowledge Base

English 简体中文繁體中文にほんご