๐ ๐ง Speech Emotion Recognition with Wav2Vec2
This project utilizes the Wav2Vec2 model to recognize emotions in speech. Its goal is to classify audio recordings into various emotional categories, such as Happy, Sad, and Surprised.
๐ Quick Start
To use the model for speech emotion recognition, you can follow these steps:
from transformers import AutoModelForAudioClassification, AutoFeatureExtractor
import librosa
import torch
import numpy as np
model_id = "firdhokk/speech-emotion-recognition-with-facebook-wav2vec2-large-xlsr-53"
model = AutoModelForAudioClassification.from_pretrained(model_id)
feature_extractor = AutoFeatureExtractor.from_pretrained(model_id, do_normalize=True, return_attention_mask=True)
id2label = model.config.id2label
def preprocess_audio(audio_path, feature_extractor, max_duration=30.0):
audio_array, sampling_rate = librosa.load(audio_path, sr=feature_extractor.sampling_rate)
max_length = int(feature_extractor.sampling_rate * max_duration)
if len(audio_array) > max_length:
audio_array = audio_array[:max_length]
else:
audio_array = np.pad(audio_array, (0, max_length - len(audio_array)))
inputs = feature_extractor(
audio_array,
sampling_rate=feature_extractor.sampling_rate,
max_length=max_length,
truncation=True,
return_attention_mask=True,
return_tensors="pt",
)
return inputs
def predict_emotion(audio_path, model, feature_extractor, id2label, max_duration=30.0):
inputs = preprocess_audio(audio_path, feature_extractor, max_duration)
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model = model.to(device)
inputs = {key: value.to(device) for key, value in inputs.items()}
with torch.no_grad():
outputs = model(**inputs)
logits = outputs.logits
predicted_id = torch.argmax(logits, dim=-1).item()
predicted_label = id2label[predicted_id]
return predicted_label
audio_path = "/content/drive/MyDrive/Audio/Speech_URDU/Happy/SM5_F4_H058.wav"
predicted_emotion = predict_emotion(audio_path, model, feature_extractor, id2label)
print(f"Predicted Emotion: {predicted_emotion}")
โจ Features
- Leverage the Wav2Vec2 model for speech emotion recognition.
- Classify audio recordings into multiple emotional categories.
- Use multiple datasets for training and evaluation.
๐ฆ Installation
The project depends on the following frameworks:
Property |
Details |
Transformers |
4.44.2 |
Pytorch |
2.4.1+cu121 |
Datasets |
3.0.0 |
Tokenizers |
0.19.1 |
You can install these dependencies according to their official documentation.
๐ Documentation
๐ Dataset
The dataset used for training and evaluation is sourced from multiple datasets, including:
The dataset contains recordings labeled with various emotions. The distribution of emotions in the dataset is as follows:
Emotion |
Count |
sad |
752 |
happy |
752 |
angry |
752 |
neutral |
716 |
disgust |
652 |
fearful |
652 |
surprised |
652 |
calm |
192 |
The "calm" emotion was excluded during training due to its underrepresentation.
๐ค Preprocessing
- Audio Loading: Use Librosa to load audio files and convert them to numpy arrays.
- Feature Extraction: Process the audio data using the Wav2Vec2 Feature Extractor, which standardizes and normalizes the audio features for input to the model.
๐ง Model
The model used is the Wav2Vec2 Large XLR-53 model, fine-tuned for audio classification tasks:
The emotion labels are mapped to numeric IDs for model training and evaluation.
โ๏ธ Training
The model is trained with the following parameters:
- Learning Rate:
5e-05
- Train Batch Size:
2
- Eval Batch Size:
2
- Random Seed:
42
- Gradient Accumulation Steps:
5
- Total Train Batch Size:
10
(effective batch size after gradient accumulation)
- Optimizer: Adam with parameters:
betas=(0.9, 0.999)
and epsilon=1e-08
- Learning Rate Scheduler:
linear
- Warmup Ratio for LR Scheduler:
0.1
- Number of Epochs:
25
- Mixed Precision Training: Native AMP (Automatic Mixed Precision)
The training uses Wandb for experiment tracking and monitoring.
๐ Metrics
The following evaluation metrics were obtained after training the model:
- Loss:
0.4989
- Accuracy:
0.9168
- Precision:
0.9209
- Recall:
0.9168
- F1 Score:
0.9166
These metrics demonstrate the model's performance on the speech emotion recognition task.
๐งช Results
After training, the model is evaluated on the test dataset, and the results are monitored using Wandb in this Link.
Training Loss |
Epoch |
Step |
Validation Loss |
Accuracy |
Precision |
Recall |
F1 |
1.9343 |
0.9995 |
394 |
1.9277 |
0.2505 |
0.1425 |
0.2505 |
0.1691 |
1.7944 |
1.9990 |
788 |
1.6446 |
0.4574 |
0.5759 |
0.4574 |
0.4213 |
1.4601 |
2.9985 |
1182 |
1.3242 |
0.5953 |
0.6183 |
0.5953 |
0.5709 |
1.0551 |
3.9980 |
1576 |
1.0764 |
0.6623 |
0.6659 |
0.6623 |
0.6447 |
0.8934 |
5.0 |
1971 |
0.9209 |
0.7059 |
0.7172 |
0.7059 |
0.6825 |
1.1156 |
5.9995 |
2365 |
0.8292 |
0.7465 |
0.7635 |
0.7465 |
0.7442 |
0.6307 |
6.9990 |
2759 |
0.6439 |
0.8043 |
0.8090 |
0.8043 |
0.8020 |
0.774 |
7.9985 |
3153 |
0.6666 |
0.7921 |
0.8117 |
0.7921 |
0.7916 |
0.5537 |
8.9980 |
3547 |
0.5111 |
0.8245 |
0.8268 |
0.8245 |
0.8205 |
0.3762 |
10.0 |
3942 |
0.5506 |
0.8306 |
0.8390 |
0.8306 |
0.8296 |
0.716 |
10.9995 |
4336 |
0.5499 |
0.8276 |
0.8465 |
0.8276 |
0.8268 |
0.5372 |
11.9990 |
4730 |
0.5463 |
0.8377 |
0.8606 |
0.8377 |
0.8404 |
0.3746 |
12.9985 |
5124 |
0.4758 |
0.8611 |
0.8714 |
0.8611 |
0.8597 |
0.4317 |
13.9980 |
5518 |
0.4438 |
0.8742 |
0.8843 |
0.8742 |
0.8756 |
0.2104 |
15.0 |
5913 |
0.4426 |
0.8803 |
0.8864 |
0.8803 |
0.8806 |
0.3193 |
15.9995 |
6307 |
0.4741 |
0.8671 |
0.8751 |
0.8671 |
0.8683 |
0.3445 |
16.9990 |
6701 |
0.3850 |
0.9037 |
0.9047 |
0.9037 |
0.9038 |
0.2777 |
17.9985 |
7095 |
0.4802 |
0.8834 |
0.8923 |
0.8834 |
0.8836 |
0.4406 |
18.9980 |
7489 |
0.4053 |
0.9047 |
0.9096 |
0.9047 |
0.9043 |
0.1707 |
20.0 |
7884 |
0.4434 |
0.9067 |
0.9129 |
0.9067 |
0.9069 |
0.2138 |
20.9995 |
8278 |
0.5051 |
0.9037 |
0.9155 |
0.9037 |
0.9053 |
0.1812 |
21.9990 |
8672 |
0.4238 |
0.8955 |
0.9007 |
0.8955 |
0.8953 |
0.3639 |
22.9985 |
9066 |
0.4021 |
0.9138 |
0.9182 |
0.9138 |
0.9143 |
0.3193 |
23.9980 |
9460 |
0.4989 |
0.9168 |
0.9209 |
0.9168 |
0.9166 |
0.2067 |
24.9873 |
9850 |
0.4959 |
0.8976 |
0.9032 |
0.8976 |
0.8975 |
๐ License
The project is licensed under the apache-2.0
license.