Roberta Base Go Emotions

🚀 RoBERTa-base for Go Emotions Multi-label Classification

This model is trained on the Go Emotions dataset for multi-label text classification, offering 28 'probability' float outputs for each input text.

🚀 Quick Start

This model can be easily used in Huggingface Transformers. The simplest way is to use a pipeline:

from transformers import pipeline

classifier = pipeline(task="text-classification", model="SamLowe/roberta-base-go_emotions", top_k=None)

sentences = ["I am not having a great day"]

model_outputs = classifier(sentences)
print(model_outputs[0])
# produces a list of dicts for each of the labels

✨ Features

• Multi-label Classification: Capable of classifying text into one or multiple emotions from 28 available labels.
• ONNX Support: An ONNX version (including an INT8 quantized ONNX version) is available, which offers faster inference, especially for smaller batch sizes, reduces dependency size, and enhances multi-platform compatibility.
• Trained on Real-world Data: The model is trained on the Go Emotions dataset, which is based on Reddit data, making it suitable for real-world text classification tasks.

📦 Installation

No specific installation steps are provided in the original document.

📚 Documentation

Overview

The model is trained from roberta-base on the go_emotions dataset for multi-label classification.

ONNX Version

A version of this model in ONNX format (including an INT8 quantized ONNX version) is now available at https://huggingface.co/SamLowe/roberta-base-go_emotions-onnx. These are faster for inference, especially for smaller batch sizes, massively reduce the size of the dependencies required for inference, make inference of the model more multi-platform, and in the case of the quantized version reduce the model file/download size by 75% whilst retaining almost all the accuracy if you only need inference.

Dataset

go_emotions is based on Reddit data and has 28 labels. It is a multi-label dataset where one or multiple labels may apply for any given input text. This model outputs 28 'probability' float values for any given input text. Typically, a threshold of 0.5 is applied to the probabilities for the prediction of each label.

Model Creation

The model was trained using AutoModelForSequenceClassification.from_pretrained with problem_type="multi_label_classification" for 3 epochs with a learning rate of 2e-5 and weight decay of 0.01.

Evaluation / Metrics

Evaluation of the model is available at

• https://github.com/samlowe/go_emotions-dataset/blob/main/eval-roberta-base-go_emotions.ipynb

Summary

Using the dataset test split to evaluate the multi-label output (binarized via a threshold of 0.5), the following metrics are obtained:

• Accuracy: 0.474
• Precision: 0.575
• Recall: 0.396
• F1: 0.450

However, considering the multi-label nature of the dataset and the different representations of labels, evaluating metrics per label is more meaningful.

With a threshold of 0.5 applied to binarize the model outputs:

Optimizing the threshold per label for the one that gives the optimum F1 metrics:

This improves the overall metrics:

• Precision: 0.542
• Recall: 0.577
• F1: 0.541

Or if calculated weighted by the relative size of the support of each label:

• Precision: 0.572
• Recall: 0.677
• F1: 0.611

Commentary on the Dataset

Some labels (e.g., gratitude) perform very strongly with F1 exceeding 0.9 when considered independently, while others (e.g., relief) perform very poorly.

This is a challenging dataset. Labels such as relief have much fewer examples in the training data (less than 100 out of the 40k+, and only 11 in the test split).

There is also some ambiguity and/or labelling errors visible in the training data of go_emotions, which is suspected to constrain the performance. Data cleaning on the dataset to reduce some of the mistakes, ambiguity, conflicts, and duplication in the labelling would produce a higher performing model.

🔧 Technical Details

The model was trained using AutoModelForSequenceClassification.from_pretrained with problem_type="multi_label_classification" for 3 epochs with a learning rate of 2e-5 and weight decay of 0.01.

📄 License

This project is licensed under the MIT License.