Response Quality Classifier Large

🚀 Response Quality Classifier Large

This classification model is designed to evaluate the relevance and specificity of the last message in a dialogue context.

🚀 Quick Start

This classification model is based on sberbank-ai/ruRoberta-large. It is used to assess the relevance and specificity of the last message within a dialogue context.

✨ Features

• Relevance Assessment: Determines whether the last message in the dialogue is relevant to the entire dialogue context.
• Specificity Evaluation: Checks if the last message is interesting and promotes the continuation of the dialogue.

📦 Installation

No specific installation steps are provided in the original document.

💻 Usage Examples

Basic Usage

import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification

tokenizer = AutoTokenizer.from_pretrained('tinkoff-ai/response-quality-classifier-large')
model = AutoModelForSequenceClassification.from_pretrained('tinkoff-ai/response-quality-classifier-large')
inputs = tokenizer('[CLS]привет[SEP]привет![SEP]как дела?[RESPONSE_TOKEN]норм, у тя как?', max_length=128, add_special_tokens=False, return_tensors='pt')
with torch.inference_mode():
    logits = model(**inputs).logits
    probas = torch.sigmoid(logits)[0].cpu().detach().numpy()
relevance, specificity = probas

Advanced Usage

The basic usage example covers the main functionality. There is no additional advanced usage information provided in the original document.

📚 Documentation

The labels are explained as follows:

• relevance: Indicates whether the last message in the dialogue is relevant in the context of the full dialogue.
• specificity: Shows if the last message in the dialogue is interesting and promotes the continuation of the dialogue.

It is pretrained on a large corpus of dialog data in an unsupervised manner. The model is trained to predict whether the last response was from a real dialog or randomly selected from another dialog. Then it was finetuned on manually labelled examples (the dataset will be posted soon).

The model was trained with three messages in the context and one response. Each message was tokenized separately with max_length = 32.

The performance of the model on the validation split (the dataset will be posted soon) (with the best thresholds for validation samples) is as follows:

🔧 Technical Details

The model's training process involves unsupervised pretraining on a large dialog corpus and then finetuning on manually labelled examples. The tokenization process uses a max_length of 32 for each message.

📄 License

The model is licensed under the MIT license.

Additional Information

Here are some example dialogs used to demonstrate the model:

• Dialog example 1: "[CLS]привет[SEP]привет![SEP]как дела?[RESPONSE_TOKEN]супер, вот только проснулся, у тебя как?"
• Dialog example 2: "[CLS]привет[SEP]привет![SEP]как дела?[RESPONSE_TOKEN]норм"
• Dialog example 3: "[CLS]привет[SEP]привет![SEP]как дела?[RESPONSE_TOKEN]норм, у тя как?"

You can easily interact with this model through the app.

The work was done during an internship at Tinkoff by egoriyaa, mentored by solemn-leader.