One-for-All Toxicity V3 Open-Source Text Toxicity Detection Model - Supports 55 Languages for Identifying Harmful and Spam Content

Home

One For All Toxicity V3

Developed by FredZhang7

Multilingual text toxicity detection model supporting 55 languages for identifying harmful or spam content

Text Classification

Transformers

Supports Multiple Languages#Multilingual Toxicity Detection #High-precision Text Moderation #Short Text Optimization

Downloads 570

Release Time : 6/29/2023

Model Overview

BERT-based multilingual text classification model specifically designed for toxicity detection in content moderation scenarios, capable of identifying harmful text content in multiple languages

Model Features

Multilingual Support

Supports toxicity detection in 55 languages, including major and some niche languages

High Accuracy

Training accuracy reaches 99.5% for English and 98.6% for other languages, with final validation accuracy of 96.8%

Optimized Short Text Detection

Improved short text classification accuracy through manually annotated supplementary training data

Efficient Architecture

Optimized based on bert-base-multilingual-cased, delivering excellent performance under limited resources

Model Capabilities

Multilingual Text Classification

Harmful Content Identification

Spam Content Detection

Content Moderation Assistance

Use Cases

Content Moderation

Social Media Content Filtering

Automatically identifies harmful information in user-generated content

Effectively reduces manual moderation workload

Multilingual Forum Management

Detects spam or inappropriate content in multiple languages

Supports real-time detection in 55 languages

Cybersecurity

Cyberbullying Prevention

Identifies offensive language in chats and comments

Helps create safer online environments

🚀 One-for-all Toxicity v3

A multilingual text classification model for toxicity detection, supporting 55 languages and achieving high accuracy.

🚀 Quick Start

You can find the link to the distilbert spam defender here. The v1 (TensorFlow) model in SavedModel format can be found on this page. The license for the v1 model is Apache 2.0.

📚 Documentation

Model Comparison

Property	v3	v1
Base Model	bert-base-multilingual-cased	nlpaueb/legal-bert-small-uncased
Base Tokenizer	bert-base-multilingual-cased	bert-base-multilingual-cased
Framework	PyTorch	TensorFlow
Dataset Size	3.0M	2.68M
Train Split	80% English 20% English + 100% Multilingual	None
English Train Accuracy	99.5%	N/A (≈97.5%)
Other Train Accuracy	98.6%	96.6%
Final Val Accuracy	96.8%	94.6%
Languages	55	N/A (≈35)
Hyperparameters	maxlen=208 padding='max_length' batch_size=112 optimizer=AdamW learning_rate=1e-5 loss=BCEWithLogitsLoss()	maxlen=192 padding='max_length' batch_size=16 optimizer=Adam learning_rate=1e-5 loss="binary_crossentropy"
Training Stopped	7/20/2023	9/05/2022

Model Improvement

I manually annotated more data on top of Toxi Text 3M and added them to the training set. Training on Toxi Text 3M alone results in a biased model that classifies short text with lower precision.

Model Selection

Models tested for v2: roberta, xlm-roberta, bert-small, bert-base-cased/uncased, bert-multilingual-cased/uncased, and alberta-large-v2. Of these, I chose bert-multilingual-cased because it performs better with the same amount of resources as the others for this particular task.

💻 Usage Examples

Basic Usage

text = "hello world!"

import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
tokenizer = AutoTokenizer.from_pretrained("FredZhang7/one-for-all-toxicity-v3")
model = AutoModelForSequenceClassification.from_pretrained("FredZhang7/one-for-all-toxicity-v3").to(device)

encoding = tokenizer.encode_plus(
    text,
    add_special_tokens=True,
    max_length=208,
    padding="max_length",
    truncation=True,
    return_tensors="pt"
)
print('device:', device)
input_ids = encoding["input_ids"].to(device)
attention_mask = encoding["attention_mask"].to(device)

with torch.no_grad():
    outputs = model(input_ids, attention_mask=attention_mask)
    logits = outputs.logits
    predicted_labels = torch.argmax(logits, dim=1)

print(predicted_labels)

📄 License

The license for this project is CC BY 4.0.

Attribution

If you distribute, remix, adapt, or build upon One-for-all Toxicity v3, please credit "AIstrova Technologies Inc." in your README.md, application description, research, or website.

Featured Recommended AI Models

Empowering the Future, Your AI Solution Knowledge Base

English 简体中文繁體中文にほんご