Emo Mobilebert

🚀 Emo-MobileBERT

A thin version of BERT LARGE, trained on the EmoContext Dataset from scratch

Emo-MobileBERT is a compact and efficient model based on MobileBERT, trained specifically for emotion recognition on the EmoContext dataset. It offers a balance between performance and resource consumption, making it suitable for deployment on resource - limited devices.

🚀 Quick Start

Pipelining the Model

from transformers import AutoTokenizer, AutoModelForSequenceClassification, pipeline

tokenizer = AutoTokenizer.from_pretrained("lordtt13/emo-mobilebert")

model = AutoModelForSequenceClassification.from_pretrained("lordtt13/emo-mobilebert")

nlp_sentence_classif = transformers.pipeline('sentiment-analysis', model = model, tokenizer = tokenizer)
nlp_sentence_classif("I've never had such a bad day in my life")
# Output: [{'label': 'sad', 'score': 0.93153977394104}]

✨ Features

Details of MobileBERT

The MobileBERT model was presented in MobileBERT: a Compact Task - Agnostic BERT for Resource - Limited Devices by Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, Denny Zhou. Here is the abstract:

Natural Language Processing (NLP) has recently achieved great success by using huge pre - trained models with hundreds of millions of parameters. However, these models suffer from heavy model sizes and high latency such that they cannot be deployed to resource - limited mobile devices. In this paper, we propose MobileBERT for compressing and accelerating the popular BERT model. Like the original BERT, MobileBERT is task - agnostic, that is, it can be generically applied to various downstream NLP tasks via simple fine - tuning. Basically, MobileBERT is a thin version of BERT_LARGE, while equipped with bottleneck structures and a carefully designed balance between self - attentions and feed - forward networks. To train MobileBERT, we first train a specially designed teacher model, an inverted - bottleneck incorporated BERT_LARGE model. Then, we conduct knowledge transfer from this teacher to MobileBERT. Empirical studies show that MobileBERT is 4.3x smaller and 5.5x faster than BERT_BASE while achieving competitive results on well - known benchmarks. On the natural language inference tasks of GLUE, MobileBERT achieves a GLUE score of 77.7 (0.6 lower than BERT_BASE), and 62 ms latency on a Pixel 4 phone. On the SQuAD v1.1/v2.0 question answering task, MobileBERT achieves a dev F1 score of 90.0/79.2 (1.5/2.1 higher than BERT_BASE).

Details of the downstream task (Emotion Recognition) - Dataset 📚

SemEval - 2019 Task 3: EmoContext Contextual Emotion Detection in Text

In this dataset, given a textual dialogue i.e. an utterance along with two previous turns of context, the goal was to infer the underlying emotion of the utterance by choosing from four emotion classes:

• sad 😢
• happy 😃
• angry 😡
• others

📦 Installation

The installation steps are not provided in the original README. If you want to use this model, you can follow the general installation steps of the transformers library:

pip install transformers

📚 Documentation

Model training

The training script is present here.

📄 License

The license information is not provided in the original README.

Created by Tanmay Thakur | LinkedIn