Hubert Large Superb Er

🚀 Hubert-Large for Emotion Recognition

This is a Hubert-Large model designed for emotion recognition, leveraging pre - trained features to classify emotions in speech.

🚀 Quick Start

This model can be used for emotion recognition in speech. You can utilize it through the Audio Classification pipeline or use it directly.

✨ Features

• Ported Model: It is a ported version of S3PRL's Hubert for the SUPERB Emotion Recognition task.
• Pre - trained Base: The base model is hubert - large - ll60k, pretrained on 16kHz sampled speech audio.
• Standard Dataset: Adopts the widely used IEMOCAP dataset for emotion recognition and follows the conventional evaluation protocol.

📦 Installation

No specific installation steps are provided in the original document, so this section is skipped.

💻 Usage Examples

Basic Usage

You can use the model via the Audio Classification pipeline:

from datasets import load_dataset
from transformers import pipeline

dataset = load_dataset("anton-l/superb_demo", "er", split="session1")

classifier = pipeline("audio-classification", model="superb/hubert-large-superb-er")
labels = classifier(dataset[0]["file"], top_k=5)

Advanced Usage

Or use the model directly:

import torch
import librosa
from datasets import load_dataset
from transformers import HubertForSequenceClassification, Wav2Vec2FeatureExtractor

def map_to_array(example):
    speech, _ = librosa.load(example["file"], sr=16000, mono=True)
    example["speech"] = speech
    return example

# load a demo dataset and read audio files
dataset = load_dataset("anton-l/superb_demo", "er", split="session1")
dataset = dataset.map(map_to_array)

model = HubertForSequenceClassification.from_pretrained("superb/hubert-large-superb-er")
feature_extractor = Wav2Vec2FeatureExtractor.from_pretrained("superb/hubert-large-superb-er")

# compute attention masks and normalize the waveform if needed
inputs = feature_extractor(dataset[:4]["speech"], sampling_rate=16000, padding=True, return_tensors="pt")

logits = model(**inputs).logits
predicted_ids = torch.argmax(logits, dim=-1)
labels = [model.config.id2label[_id] for _id in predicted_ids.tolist()]

📚 Documentation

Model description

This is a ported version of S3PRL's Hubert for the SUPERB Emotion Recognition task. The base model is hubert - large - ll60k, which is pretrained on 16kHz sampled speech audio. When using the model make sure that your speech input is also sampled at 16Khz.

For more information refer to SUPERB: Speech processing Universal PERformance Benchmark

Task and dataset description

Emotion Recognition (ER) predicts an emotion class for each utterance. The most widely used ER dataset IEMOCAP is adopted, and we follow the conventional evaluation protocol: we drop the unbalanced emotion classes to leave the final four classes with a similar amount of data points and cross - validate on five folds of the standard splits.

For the original model's training and evaluation instructions refer to the S3PRL downstream task README.

🔧 Technical Details

No detailed technical implementation details are provided in the original document, so this section is skipped.

📄 License

This project is licensed under the Apache 2.0 license.

Eval results

The evaluation metric is accuracy.

BibTeX entry and citation info

@article{yang2021superb,
  title={SUPERB: Speech processing Universal PERformance Benchmark},
  author={Yang, Shu - wen and Chi, Po - Han and Chuang, Yung - Sung and Lai, Cheng - I Jeff and Lakhotia, Kushal and Lin, Yist Y and Liu, Andy T and Shi, Jiatong and Chang, Xuankai and Lin, Guan - Ting and others},
  journal={arXiv preprint arXiv:2105.01051},
  year={2021}
}