Voice Safety Classifier

🚀 Voice Safety Classifier

A large classification model trained on real - world data for toxicity detection and classification.

🚀 Quick Start

This model can serve as a new benchmark for advancing research in toxicity detection and classification. It starts with the original weights from WavLM base plus and is fine - tuned for multilabel classification.

✨ Features

• Multilabel Classification: A single output can have multiple labels. The model outputs an n by 6 output tensor with inferred labels such as Profanity, DatingAndSexting, Racist, Bullying, Other, NoViolation.
• Automated Labeling: Audio clips are automatically labeled using a synthetic data pipeline described in our blog post.
• High Precision: When treated as a binary classifier across 4 toxicity classes, it achieves a binarized average precision of 94.48%.

📦 Installation

The dependencies for the inference file can be installed as follows:

pip install -r requirements.txt

💻 Usage Examples

Basic Usage

The inference file contains useful helper functions to preprocess the audio file for proper inference. To run the inference file, please run the following command:

python inference.py --audio_file <your audio file path> --model_path <path to Huggingface model>

You can get the model weights either by downloading from the model releases page here, or from HuggingFace under roblox/voice-safety-classifier. If model_path isn’t specified, the model will be loaded directly from HuggingFace.

📚 Documentation

Model Training

We started with the original weights from the WavLM base plus and fine - tuned it with 2,374 hours of voice chat audio clips for multilabel classification.

Model Evaluation

We evaluated this model on a data set with human - annotated labels that contained a total of 9,795 samples. The class distribution is shown in the table below. Note that we did not include the "other" category in this evaluation data set.

If we set the same threshold across all classes and treat the model as a binary classifier across all 4 toxicity classes (Profanity, DatingAndSexting, Racist, Bullying), we get a binarized average precision of 94.48%. The precision - recall curve is as shown below.

📄 License

This project is licensed under the CC BY - SA 3.0 license.