Model Parameters

Input Format

Output Format

Considerations

Intended Users

• Text Generation Researchers and Developers

Use Cases

• User Experience Monitoring: The classification model can be used to monitor conversations in real-time to detect any toxic behavior by users. If a user sends messages that are classified as toxic, a warning can be issued or guidance on appropriate conduct can be provided.

• Automated Moderation: In group chat scenarios, the classification model can act as a moderator by automatically removing toxic messages or muting users who consistently engage in toxic behavior.

• Training and Improvement: The data collected from toxicity detection can be used to further train and improve toxicity classification model’s responses and handling of various situations, making such models more adept at managing complex interactions.

• Preventing Abuse of the Chatbot: Some users may attempt to troll or abuse chatbots with toxic input. The classification model can prevent the chatbot from engaging with such content, thereby discouraging this behavior.

Ethical Considerations

• Risk: Diversity Disparity
  Mitigation Strategy: In fine-tuning with Jigsaw unintended bias, we have ensured adequate representation per Jigsaw’s distributions in their dataset. Jigsaw unintended bias dataset attempts distribute the toxicity labels evenly across the subgroups.

• Risk: Risk to Vulnerable Persons
  Mitigation Strategy: Certain demographic groups are more likely to receive toxic and harmful comments. Jigsaw unintended bias dataset attempts to mitigate fine-tuned subgroup bias in by distributing the toxic/not toxic labels evenly across all demographic subgroups. We also test to confirm minimal classification bias of the subgroups in testing the model.

Quantitative Analysis:

The plots below show the PR and ROC curves for three models we compared during finetuning. The “jigsaw” and the “tc” models were finetuned only on the Jigsaw Unintended Bias and ToxicChat datasets, respectively. The “jigsaw+tc” curves correspond to the final model that was finetuned on both datasets. Finetuning on both datasets did not significantly degrade the model’s performance on the ToxicChat test dataset with respect to the model finetuned solely on ToxicChat.

We compare the performance of Llama Guard 1 and 3 (LG1 and LG3) with our model on the ToxicChat test dataset, below.

* from LG paper: https://arxiv.org/abs/2312.06674

Note that Llama Guard was not finetuned on ToxicChat. However, from the LG1 paper, they reported an AUPRC of ~.81 when they finetuned LLama Guard 1 on ToxicChat. Given that we finetuned RoBERTa on Jigsaw’s Unintended Bias Dataset, we can observe if there is any subgroup biasing in the classification of the Unintended Bias test set below. These metrics were computed using Intel/bias_auc.

*Only subgroups with at least 500 examples in the test dataset are shown.