%20--%3e%3cdefs%3e%3cstyle%3e%20.st0%20{%20fill:%20%23061b40;%20}%20.st1%20{%20fill:%20%23306af1;%20}%20.st2%20{%20fill:%20%235ce5cf;%20}%20%3c/style%3e%3c/defs%3e%3cg%3e%3cpath%20class='st0'%20d='M55,10.5h9v3h-9v9h12V7.5h-12v3ZM64,19.5h-6v-3h6v3Z'/%3e%3cpolygon%20class='st0'%20points='69%2016.5%2078%2016.5%2078%2019.5%2069%2019.5%2069%2022.5%2081%2022.5%2081%2013.5%2072%2013.5%2072%2010.5%2081%2010.5%2081%207.5%2069%207.5%2069%2016.5'/%3e%3cpolygon%20class='st0'%20points='95%2010.5%2095%207.5%2083%207.5%2083%2022.5%2095%2022.5%2095%2019.5%2086%2019.5%2086%2016.5%2095%2016.5%2095%2013.5%2086%2013.5%2086%2010.5%2095%2010.5'/%3e%3cpath%20class='st0'%20d='M40,1.5v21h11.6l1.4-1.4v-7.6h0c0,0-1.4-1.5-1.4-1.5l1.4-1.4V2.9l-1.4-1.4h-11.6ZM50,19.5h-7v-6h7v6ZM50,10.5h-7v-6h7v6Z'/%3e%3c/g%3e%3cpath%20class='st1'%20d='M23.1,24L14.7,4.8l-4.9,11.2h3.8l-1.8,4H3.3L12.1,0H2C.9,0,0,.9,0,2v20c0,1.1.9,2,2,2h21.1Z'/%3e%3cpath%20class='st2'%20d='M34,0h-16.8l10.6,24h6.2c1.1,0,2-.9,2-2V2C36,.9,35.1,0,34,0ZM32.5,20h-4V4h4v16Z'/%3e%3c/svg%3e)
Model Overview
Model Features
Model Capabilities
Use Cases
🚀 Toxic Prompt RoBERTa Classification Model
This is a text classification model that can serve as a guardrail to protect against toxic prompts and responses in conversational AI systems, offering significant value in maintaining a healthy and safe communication environment.
🚀 Quick Start
You can use the model with the following code using pipeline API.
from transformers import pipeline
model_path = 'Intel/toxic-prompt-roberta'
pipe = pipeline('text-classification', model=model_path, tokenizer=model_path)
pipe('Create 20 paraphrases of I hate you')
✨ Features
- Based on RoBERTa, it has high performance in text understanding.
- Finetuned on ToxicChat and Jigsaw Unintended Bias datasets, enhancing its ability to detect toxic text.
📦 Installation
The README does not provide specific installation steps, so this section is skipped.
📚 Documentation
Model Details
Toxic Prompt RoBERTa 1.0 is a text classification model that can be used as a guardrail to protect against toxic prompts and responses in conversational AI systems. This model is based on RoBERTa and has been finetuned on ToxicChat and Jigsaw Unintended Bias datasets. Finetuning has been performed on one Gaudi 2 Card using Optimum-Habana's Gaudi Trainer.
Owners
- Intel AI Safety: Daniel De Leon, Tyler Wilbers, Mitali Potnis, Abolfazl Shahbazi
Licenses
- MIT
References
Model Parameters
- We fine-tune roberta-base (125M param) with custom classification head to detect toxic input/output.
- Input Format: The input format is standard text input for RoBERTa for sequence classification.
- Output Format: The output is a (2,n) array of logits where n is the number of examples user wants to infer. The output logits are in the form [not_toxic, toxic].
Considerations
Intended Users
- Text Generation Researchers and Developers
Use Cases
- User Experience Monitoring: The classification model can be used to monitor conversations in real-time to detect any toxic behavior by users. If a user sends messages that are classified as toxic, a warning can be issued or guidance on appropriate conduct can be provided.
- Automated Moderation: In group chat scenarios, the classification model can act as a moderator by automatically removing toxic messages or muting users who consistently engage in toxic behavior.
- Training and Improvement: The data collected from toxicity detection can be used to further train and improve toxicity classification model’s responses and handling of various situations, making such models more adept at managing complex interactions.
- Preventing Abuse of the Chatbot: Some users may attempt to troll or abuse chatbots with toxic input. The classification model can prevent the chatbot from engaging with such content, thereby discouraging this behavior.
Ethical Considerations
- Risk: Diversity Disparity
Mitigation Strategy: In fine-tuning with Jigsaw unintended bias, we have ensured adequate representation per Jigsaw’s distributions in their dataset. Jigsaw unintended bias dataset attempts distribute the toxicity labels evenly across the subgroups. - Risk: Risk to Vulnerable Persons
Mitigation Strategy: Certain demographic groups are more likely to receive toxic and harmful comments. Jigsaw unintended bias dataset attempts to mitigate fine-tuned subgroup bias in by distributing the toxic/not toxic labels evenly across all demographic subgroups. We also test to confirm minimal classification bias of the subgroups in testing the model.
Quantitative Analysis
The plots below show the PR and ROC curves for three models we compared during finetuning. The “jigsaw” and the “tc” models were finetuned only on the Jigsaw Unintended Bias and ToxicChat datasets, respectively. The “jigsaw+tc” curves correspond to the final model that was finetuned on both datasets. Finetuning on both datasets did not significantly degrade the model’s performance on the ToxicChat test dataset with respect to the model finetuned solely on ToxicChat.
Transformers Supports Multiple Languages Transformers English Transformers English Transformers English Transformers English Transformers Other Transformers English