Speech Emotion Recognition With Openai Whisper Large V3

🚀 Speech Emotion Recognition with Whisper

This project uses the Whisper model to recognize emotions in speech. It aims to classify audio recordings into different emotional categories, such as Happy, Sad, and Surprised.

🚀 Quick Start

To use this speech emotion recognition model, you can follow these steps:

from transformers import AutoModelForAudioClassification, AutoFeatureExtractor
import librosa
import torch
import numpy as np

model_id = "firdhokk/speech-emotion-recognition-with-openai-whisper-large-v3"
model = AutoModelForAudioClassification.from_pretrained(model_id)

feature_extractor = AutoFeatureExtractor.from_pretrained(model_id, do_normalize=True)
id2label = model.config.id2label

def preprocess_audio(audio_path, feature_extractor, max_duration=30.0):
    audio_array, sampling_rate = librosa.load(audio_path, sr=feature_extractor.sampling_rate)
    
    max_length = int(feature_extractor.sampling_rate * max_duration)
    if len(audio_array) > max_length:
        audio_array = audio_array[:max_length]
    else:
        audio_array = np.pad(audio_array, (0, max_length - len(audio_array)))

    inputs = feature_extractor(
        audio_array,
        sampling_rate=feature_extractor.sampling_rate,
        max_length=max_length,
        truncation=True,
        return_tensors="pt",
    )
    return inputs

def predict_emotion(audio_path, model, feature_extractor, id2label, max_duration=30.0):
    inputs = preprocess_audio(audio_path, feature_extractor, max_duration)
    
    device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
    model = model.to(device)
    inputs = {key: value.to(device) for key, value in inputs.items()}

    with torch.no_grad():
        outputs = model(**inputs)

    logits = outputs.logits
    predicted_id = torch.argmax(logits, dim=-1).item()
    predicted_label = id2label[predicted_id]
    
    return predicted_label

audio_path = "/content/drive/MyDrive/Audio/Speech_URDU/Happy/SM5_F4_H058.wav"

predicted_emotion = predict_emotion(audio_path, model, feature_extractor, id2label)
print(f"Predicted Emotion: {predicted_emotion}")

✨ Features

• Leverage the Whisper model for speech emotion recognition.
• Classify audio recordings into multiple emotional categories.
• Use multiple datasets for training and evaluation to ensure model generalization.

📦 Installation

The framework versions used in this project are as follows:

💻 Usage Examples

Basic Usage

from transformers import AutoModelForAudioClassification, AutoFeatureExtractor
import librosa
import torch
import numpy as np

model_id = "firdhokk/speech-emotion-recognition-with-openai-whisper-large-v3"
model = AutoModelForAudioClassification.from_pretrained(model_id)

feature_extractor = AutoFeatureExtractor.from_pretrained(model_id, do_normalize=True)
id2label = model.config.id2label

audio_path = "/content/drive/MyDrive/Audio/Speech_URDU/Happy/SM5_F4_H058.wav"

predicted_emotion = predict_emotion(audio_path, model, feature_extractor, id2label)
print(f"Predicted Emotion: {predicted_emotion}")

Advanced Usage

def preprocess_audio(audio_path, feature_extractor, max_duration=30.0):
    audio_array, sampling_rate = librosa.load(audio_path, sr=feature_extractor.sampling_rate)
    
    max_length = int(feature_extractor.sampling_rate * max_duration)
    if len(audio_array) > max_length:
        audio_array = audio_array[:max_length]
    else:
        audio_array = np.pad(audio_array, (0, max_length - len(audio_array)))

    inputs = feature_extractor(
        audio_array,
        sampling_rate=feature_extractor.sampling_rate,
        max_length=max_length,
        truncation=True,
        return_tensors="pt",
    )
    return inputs

def predict_emotion(audio_path, model, feature_extractor, id2label, max_duration=30.0):
    inputs = preprocess_audio(audio_path, feature_extractor, max_duration)
    
    device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
    model = model.to(device)
    inputs = {key: value.to(device) for key, value in inputs.items()}

    with torch.no_grad():
        outputs = model(**inputs)

    logits = outputs.logits
    predicted_id = torch.argmax(logits, dim=-1).item()
    predicted_label = id2label[predicted_id]
    
    return predicted_label

📚 Documentation

🗂 Dataset

The dataset used for training and evaluation is sourced from multiple datasets, including:

• RAVDESS
• SAVEE
• TESS
• URDU

The dataset contains recordings labeled with various emotions. The distribution of the emotions in the dataset is as follows:

The "calm" emotion was excluded during training due to its underrepresentation.

🎤 Preprocessing

• Audio Loading: Use Librosa to load the audio files and convert them to numpy arrays.
• Feature Extraction: Process the audio data using the Whisper Feature Extractor, which standardizes and normalizes the audio features for input to the model.

🔧 Model

The model used is the Whisper Large V3 model, fine - tuned for audio classification tasks:

• Model: openai/whisper-large-v3
• Output: Emotion labels (Angry', 'Disgust', 'Fearful', 'Happy', 'Neutral', 'Sad', 'Surprised')

The emotion labels are mapped to numeric IDs for model training and evaluation.

⚙️ Training

The model is trained with the following parameters:

• Learning Rate: 5e-05
• Train Batch Size: 2
• Eval Batch Size: 2
• Random Seed: 42
• Gradient Accumulation Steps: 5
• Total Train Batch Size: 10 (effective batch size after gradient accumulation)
• Optimizer: Adam with parameters: betas=(0.9, 0.999) and epsilon=1e-08
• Learning Rate Scheduler: linear
• Warmup Ratio for LR Scheduler: 0.1
• Number of Epochs: 25
• Mixed Precision Training: Native AMP (Automatic Mixed Precision)

The training uses Wandb for experiment tracking and monitoring.

📊 Metrics

The following evaluation metrics were obtained after training the model:

• Loss: 0.5008
• Accuracy: 0.9199
• Precision: 0.9230
• Recall: 0.9199
• F1 Score: 0.9198

These metrics demonstrate the model's performance on the speech emotion recognition task.

🧪 Results

After training, the model is evaluated on the test dataset, and the results are monitored using Wandb in this Link.

📄 License

The project uses the Apache - 2.0 license.