đ Whisper Large v3 for Voice (Sounding) Quality Classification
This model is designed for voice (sounding) quality classification. It offers a solution to accurately classify voice qualities, leveraging the power of the Whisper Large v3 base model and relevant research findings.
đ Quick Start
Download repo
git clone git@github.com:tiantiaf0627/vox-profile-release.git
Install the package
conda create -n vox_profile python=3.8
cd vox-profile-release
pip install -e .
Load the model
import torch
import torch.nn.functional as F
from src.model.voice_quality.whisper_voice_quality import WhisperWrapper
device = torch.device("cuda") if torch.cuda.is_available() else "cpu"
model = WhisperWrapper.from_pretrained("tiantiaf/whisper-large-v3-voice-quality").to(device)
model.eval()
⨠Features
- Based on Whisper Large v3: Utilizes the
openai/whisper-large-v3
base model.
- Comprehensive Voice Quality Labels: Covers various aspects of voice quality including pitch, texture, volume, clarity, and rhythm.
- Specific Metric Calculation: Reports speaker - level Macro - F1 scores with a specific sampling and averaging process.
đĻ Installation
Download repo
git clone git@github.com:tiantiaf0627/vox-profile-release.git
Install the package
conda create -n vox_profile python=3.8
cd vox-profile-release
pip install -e .
đģ Usage Examples
Basic Usage
import torch
import torch.nn.functional as F
from src.model.voice_quality.whisper_voice_quality import WhisperWrapper
device = torch.device("cuda") if torch.cuda.is_available() else "cpu"
model = WhisperWrapper.from_pretrained("tiantiaf/whisper-large-v3-voice-quality").to(device)
model.eval()
voice_quality_label_list = [
'shrill', 'nasal', 'deep',
'silky', 'husky', 'raspy', 'guttural', 'vocal-fry',
'booming', 'authoritative', 'loud', 'hushed', 'soft',
'crisp', 'slurred', 'lisp', 'stammering',
'singsong', 'pitchy', 'flowing', 'monotone', 'staccato', 'punctuated', 'enunciated', 'hesitant',
]
max_audio_length = 15 * 16000
data = torch.zeros([1, 16000]).float().to(device)[:, :max_audio_length]
logits = model(
data, return_feature=False
)
voice_quality_prob = nn.Sigmoid()(torch.tensor(logits))
voice_label = list()
threshold = 0.7
predictions = (voice_quality_prob > threshold).int().detach().cpu().numpy()[0].tolist()
for label_idx in range(len(predictions)):
if predictions[label_idx] == 1: voice_label.append(voice_quality_label_list[label_idx])
print(voice_label)
đ Documentation
Model Description
This model includes the implementation of voice quality classification described in Vox - Profile: A Speech Foundation Model Benchmark for Characterizing Diverse Speaker and Speech Traits.
Metric
Specifically, we report speaker - level Macro - F1 scores. Specifically, we randomly sampled five utterances for each speaker and repeated this stratification process 20 times. The speaker - level score is computed as the average Macro - F1 across speakers. We then report the unweighted average of speaker - level Macro - F1 scores between VoxCeleb and Expresso.
Special Note
We exclude EARS from ParaSpeechCaps due to its limited number of samples in the holdout set.
The included labels are:
[
'shrill', 'nasal', 'deep', # Pitch
'silky', 'husky', 'raspy', 'guttural', 'vocal-fry', # Texture
'booming', 'authoritative', 'loud', 'hushed', 'soft', # Volume
'crisp', 'slurred', 'lisp', 'stammering', # Clarity
'singsong', 'pitchy', 'flowing', 'monotone', 'staccato', 'punctuated', 'enunciated', 'hesitant', # Rhythm
]
Kindly cite our paper if you are using our model or find it useful in your work
@article{feng2025vox,
title={Vox-Profile: A Speech Foundation Model Benchmark for Characterizing Diverse Speaker and Speech Traits},
author={Feng, Tiantian and Lee, Jihwan and Xu, Anfeng and Lee, Yoonjeong and Lertpetchpun, Thanathai and Shi, Xuan and Wang, Helin and Thebaud, Thomas and Moro-Velazquez, Laureano and Byrd, Dani and others},
journal={arXiv preprint arXiv:2505.14648},
year={2025}
}
đ License
This model is released under the bsd - 2 - clause license.
â ī¸ Important Note
We exclude EARS from ParaSpeechCaps due to its limited number of samples in the holdout set.
đĄ Usage Tip
In practice, a larger threshold would remove some noise, but it is best to aggregate predictions per speaker.