Distilbert Base Uncased Emotion

🚀 Distilbert-base-uncased-emotion

This model uses knowledge distillation during pre - training to reduce the size of a BERT model by 40% while retaining 97% of its language understanding. It is smaller and faster than BERT and other BERT - based models, and is fine - tuned on the emotion dataset.

🚀 Quick Start

Prerequisites

Make sure you have the transformers library installed. If not, you can install it using pip install transformers.

Usage Example

from transformers import pipeline
classifier = pipeline("text - classification", model='bhadresh - savani/distilbert - base - uncased - emotion', return_all_scores=True)
prediction = classifier("I love using transformers. The best part is wide range of support and its easy to use", )
print(prediction)

"""
Output:
[[
{'label': 'sadness', 'score': 0.0006792712374590337}, 
{'label': 'joy', 'score': 0.9959300756454468}, 
{'label': 'love', 'score': 0.0009452480007894337}, 
{'label': 'anger', 'score': 0.0018055217806249857}, 
{'label': 'fear', 'score': 0.00041110432357527316}, 
{'label': 'surprise', 'score': 0.0002288572577526793}
]]
"""

✨ Features

• Knowledge Distillation: Distilbert is created with knowledge distillation during the pre - training phase, reducing the size of a BERT model by 40% while retaining 97% of its language understanding.
• Fine - Tuned on Emotion Dataset: [Distilbert - base - uncased](https://huggingface.co/distilbert - base - uncased) is fine - tuned on the emotion dataset using HuggingFace Trainer with specific hyperparameters (learning rate 2e - 5, batch size 64, num_train_epochs = 8).

📦 Installation

Ensure you have the necessary Python environment. Install the transformers library using the following command:

pip install transformers

💻 Usage Examples

Basic Usage

from transformers import pipeline
classifier = pipeline("text - classification", model='bhadresh - savani/distilbert - base - uncased - emotion', return_all_scores=True)
text = "I love using transformers. The best part is wide range of support and its easy to use"
prediction = classifier(text)
print(prediction)

📚 Documentation

Model description

Distilbert is created with knowledge distillation during the pre - training phase which reduces the size of a BERT model by 40%, while retaining 97% of its language understanding. It's smaller, faster than Bert and any other Bert - based model.

[Distilbert - base - uncased](https://huggingface.co/distilbert - base - uncased) is finetuned on the emotion dataset using HuggingFace Trainer with the following hyperparameters:

learning rate 2e - 5, 
batch size 64,
num_train_epochs = 8,

Model Performance Comparision on Emotion Dataset from Twitter

Dataset

Twitter - Sentiment - Analysis.

Training procedure

Colab Notebook

Eval results

{
    'test_accuracy': 0.938,
    'test_f1': 0.937932884041714,
    'test_loss': 0.1472451239824295,
    'test_mem_cpu_alloc_delta': 0,
    'test_mem_cpu_peaked_delta': 0,
    'test_mem_gpu_alloc_delta': 0,
    'test_mem_gpu_peaked_delta': 163454464,
    'test_runtime': 5.0164,
    'test_samples_per_second': 398.69
}

Reference

• [Natural Language Processing with Transformer By Lewis Tunstall, Leandro von Werra, Thomas Wolf](https://learning.oreilly.com/library/view/natural - language - processing/9781098103231/)

📄 License

This project is licensed under the Apache - 2.0 license.