Wav2vec LnNor IPA Ft

🚀 MultiBridge/wav2vec-LnNor-IPA-ft

This model is designed for phoneme recognition tasks. It fine-tunes the wav2vec2 base model on TIMIT and LnNor datasets, with predictions in IPA.

🚀 Quick Start

Use the code below to get started with the model.

from transformers import Wav2Vec2Processor, Wav2Vec2ForCTC
from datasets import load_dataset
import torch
 
# load model and processor
processor = Wav2Vec2Processor.from_pretrained("MultiBridge/wav2vec-LnNor-IPA-ft")
model = Wav2Vec2ForCTC.from_pretrained("MultiBridge/wav2vec-LnNor-IPA-ft")
     
# load dummy dataset and read soundfiles
ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", split="validation")
 
# tokenize
input_values = processor(ds[0]["audio"]["array"], return_tensors="pt").input_values
 
# retrieve logits
with torch.no_grad():
    logits = model(input_values).logits
 
# take argmax and decode
predicted_ids = torch.argmax(logits, dim=-1)
transcription = processor.batch_decode(predicted_ids)

# => should give ['mɪstɝkwɪltɝɪzðəəpɑslʌvðəmɪdəlklæsəzændwiɑəɡlædtəwɛlkəmhɪzɡɑspəl'] for MISTER QUILTER IS THE APOSTLE OF THE MIDDLE CLASSES AND WE ARE GLAD TO WELCOME HIS GOSPEL

✨ Features

• Automatic Phonetic Transcription: Converts raw speech into phoneme sequences.
• Speech Processing Applications: Serves as a component in speech processing pipelines or prototyping.

📦 Installation

No specific installation steps are provided in the original document, so this section is skipped.

💻 Usage Examples

Basic Usage

from transformers import Wav2Vec2Processor, Wav2Vec2ForCTC
from datasets import load_dataset
import torch
 
# load model and processor
processor = Wav2Vec2Processor.from_pretrained("MultiBridge/wav2vec-LnNor-IPA-ft")
model = Wav2Vec2ForCTC.from_pretrained("MultiBridge/wav2vec-LnNor-IPA-ft")
     
# load dummy dataset and read soundfiles
ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", split="validation")
 
# tokenize
input_values = processor(ds[0]["audio"]["array"], return_tensors="pt").input_values
 
# retrieve logits
with torch.no_grad():
    logits = model(input_values).logits
 
# take argmax and decode
predicted_ids = torch.argmax(logits, dim=-1)
transcription = processor.batch_decode(predicted_ids)

# => should give ['mɪstɝkwɪltɝɪzðəəpɑslʌvðəmɪdəlklæsəzændwiɑəɡlædtəwɛlkəmhɪzɡɑspəl'] for MISTER QUILTER IS THE APOSTLE OF THE MIDDLE CLASSES AND WE ARE GLAD TO WELCOME HIS GOSPEL

📚 Documentation

Model Details

Uses

• Automatic phonetic transcription: Converting raw speech into phoneme sequences.
• Speech processing applications: Serving as a component in speech processing pipelines or prototyping.

Bias, Risks, and Limitations

• Data Specificity: By excluding recordings shorter than 2 seconds or longer than 30 seconds, and labels with fewer than 5 phonemes, some natural speech variations are ignored. This might affect the model's performance in real-world applications. The model's performance is influenced by the characteristics of TIMIT and LnNor datasets. This can lead to potential biases, especially if the target application involves speakers or dialects not well-represented in these datasets. LnNor contains non-native speech and automatically generated annotations that don't reflect true pronunciation rather canonical pronunciation. This could result in a model that fails to accurately predict non-native speech.
• Frozen Encoder: Freezing the encoder retains useful pre-learned features but also prevents the model from adapting fully to the new datasets.

Recommendations

Evaluate the model's performance for your specific use case.

Training Details

Training Data

• TIMIT: A widely-used dataset for phonetic transcription, providing a standard benchmark in speech research.
• LnNor: A multilingual dataset of high-quality speech recordings in Norwegian, English, and Polish. The dataset compiled from non-native speakers with various language proficiencies. The phoneme annotations in LnNor were generated using the WebMAUS tool, meaning they represent canonical phonemes rather than the true pronunciations typical of spontaneous speech or non native pronunciation.

Training Procedure

The original, pre-trained encoder representations were preserved - the encoder was kept frozen during fine-tuning in order to minimizes training time and resource consumption. The model was trained with CTC loss and AdamW optimizer, with no learning rate scheduler.

Preprocessing

The training dataset was filtered. Recordings shorter than 2 seconds or longer than 30 seconds were removed. Any labels consisting of fewer than 5 phonemes were discarded.

Training Hyperparameters

Training regime:

• learning rate: 1e-5
• optimizer: AdamW
• batch size: 64
• weight decay: 0.001
• epochs: 40

Speeds, Sizes, Times

• Avg epoch training time: 650s
• Number of updates: ~25k
• Final training loss: 0.09713
• Final validation loss: 0.2142

Evaluation

Testing Data, Factors & Metrics

Testing Data

The model was evaluated on TIMIT's test split.

Metrics

CER/PER (Phoneme Error Rate)

Results

PER (Phoneme Error Rate) on TIMIT's test split: 0.0416

Environmental Impact

Carbon emissions can be estimated using the Machine Learning Impact calculator presented in Lacoste et al. (2019).

• Hardware Type: Nvidia A100 - 80
• Hours used: [More Information Needed]
• Cloud Provider: Poznan University of Technology
• Compute Region: Poland
• Carbon Emitted: [More Information Needed]

Technical Specifications

Model Architecture and Objective

Transformer model + CTC loss

Compute Infrastructure

Hardware

2 x Nvidia A100 - 80

Software

• python 3.12
• transformers 4.50.0
• torch 2.6.0

Citation

BibTeX:

@article{magdalena2024lnnor,
  title={The LnNor Corpus: A spoken multilingual corpus of non-native and native Norwegian, English and Polish (Part 1)},
  author={Magdalena, Wrembel and Hwaszcz, Krzysztof and Agnieszka, Pludra and Ska{\l}ba, Anna and Weckwerth, Jaros{\l}aw and Walczak, Angelika and Sypia{\'n}ska, Jolanta and {\.Z}ychli{\'n}ski, Sylwiusz and Malarski, Kamil and K{\k{e}}dzierska, Hanna and others},
  year={2024},
  publisher={Adam Mickiewicz University}
}

@article{wrembel2024lnnor,
  title={The LnNor Corpus: A spoken multilingual corpus of non-native and native Norwegian, English and Polish--Part 2},
  author={Wrembel, Magdalena and Hwaszcz, Krzysztof and Pludra, Agnieszka and Ska{\l}ba, Anna and Weckwerth, Jaros{\l}aw and Malarski, Kamil and Cal, Zuzanna Ewa and K{\k{e}}dzierska, Hanna and Czarnecki-Verner, Tristan and Balas, Anna and others},
  year={2024},
  publisher={Adam Mickiewicz University}
}

Model Card Authors

Agnieszka Pludra Izabela Krysińska Piotr Kabaciński

Model Card Contact

• agnieszka.pludra@pearson.com
• izabela.krysinska@pearson.com
• piotr.kabacinski@pearson.com

🔧 Technical Details

The model is a Transformer-based model fine-tuned from facebook/wav2vec2-base for phoneme recognition tasks. It uses CTC loss and AdamW optimizer during training. The encoder is frozen to retain pre-learned features and reduce training time and resource consumption. The training data comes from TIMIT and LnNor datasets, with specific filtering applied to the training dataset.

📄 License

cc-by-4.0