Roberta Large Llm Content Detector

🚀 SuperAnnotate

A fine-tuned RoBERTa Large model for detecting LLM-generated text, useful in various fields like education and research.

🚀 Quick Start

To start using the SuperAnnotate LLM Content Detector, you need to install the generated_text_detector library. Run the following command:

pip install git+https://github.com/superannotateai/generated_text_detector.git@v1.0.0

✨ Features

• Text Detection: Designed to detect generated/synthetic text, which is crucial for determining text authorship, ensuring the integrity of training data, and detecting fraud in scientific and educational areas.
• Custom Architecture: Based on pre-trained RoBERTa, with a custom architecture for binary sequence classification and a single output label.
• Calibrated Model: During training, efforts were made to avoid overfitting and obtain a well - calibrated predictor.

📦 Installation

Install the generated_text_detector library using the following command:

pip install git+https://github.com/superannotateai/generated_text_detector.git@v1.0.0

💻 Usage Examples

Basic Usage

from generated_text_detector.utils.model.roberta_classifier import RobertaClassifier
from transformers import AutoTokenizer
import torch.nn.functional as F


model = RobertaClassifier.from_pretrained("SuperAnnotate/roberta-large-llm-content-detector")
tokenizer = AutoTokenizer.from_pretrained("SuperAnnotate/roberta-large-llm-content-detector")

text_example = "It's not uncommon for people to develop allergies or intolerances to certain foods as they get older. It's possible that you have always had a sensitivity to lactose (the sugar found in milk and other dairy products), but it only recently became a problem for you. This can happen because our bodies can change over time and become more or less able to tolerate certain things. It's also possible that you have developed an allergy or intolerance to something else that is causing your symptoms, such as a food additive or preservative. In any case, it's important to talk to a doctor if you are experiencing new allergy or intolerance symptoms, so they can help determine the cause and recommend treatment."

tokens = tokenizer.encode_plus(
    text_example,
    add_special_tokens=True,
    max_length=512,
    padding='longest',
    truncation=True,
    return_token_type_ids=True,
    return_tensors="pt"
)

_, logits = model(**tokens)

proba = F.sigmoid(logits).squeeze(1).item()

print(proba)

📚 Documentation

Model Details

Model Description

Model Sources

• Repository: GitHub for HTTP service

Training Data

The training data was sourced from two open datasets with different proportions and underwent filtering:

1. HC3 | 63%
2. IDMGSP | 37%

The training dataset contained approximately 20k pairs of text - label with an approximate balance of classes. The texts in the dataset follow a logical structure where human - written and model - generated texts refer to a single prompt/instruction, though the prompts themselves were not used during training.

⚠️ Important Note Key n - grams (n ranging from 2 to 5) that exhibited the highest correlation with target labels were identified and removed from the training data using the chi - squared test.

Peculiarity

During training, one of the priorities was not only maximizing the quality of predictions but also avoiding overfitting and obtaining an adequately confident predictor. The model calibration result is shown in the following plot:

Training Details

A custom architecture was chosen for its ability to perform binary classification while providing a single model output, as well as for its customizable settings for smoothing integrated into the loss function.

Training Arguments:

• Base Model: FacebookAI/roberta-large
• Epochs: 10
• Learning Rate: 5e - 04
• Weight Decay: 0.05
• Label Smoothing: 0.1
• Warmup Epochs: 4
• Optimizer: SGD
• Scheduler: Linear schedule with warmup

Performance

The model was evaluated on a benchmark consisting of a holdout subset of training data and a closed subset of SuperAnnotate data. The benchmark comprises 1k samples, with 200 samples per category. The model's performance is compared with open - source solutions and popular API detectors in the following table:

📄 License

The model is licensed under SAIPL.