Wav2vec2 Base Fi Voxpopuli V2 Finetuned

🚀 Wav2Vec2-base-fi-voxpopuli-v2 for Finnish ASR

This acoustic model is a fine-tuned version of facebook/wav2vec2-base-fi-voxpopuli-v2 for Finnish Automatic Speech Recognition (ASR), offering high - quality speech - to - text conversion.

✨ Features

• Fine - tuned for Finnish: Based on the pre - trained facebook/wav2vec2-base-fi-voxpopuli-v2, fine - tuned with 276.7 hours of Finnish transcribed speech data.
• Including Language Model: The repository also includes a Finnish KenLM language model for use in the decoding phase with the acoustic model.

📚 Documentation

Model description

Wav2vec2-base-fi-voxpopuli-v2 is a pretrained model by Facebook AI for Finnish speech. It is pretrained on 14.2k hours of unlabeled Finnish speech from the VoxPopuli V2 dataset using the wav2vec 2.0 objective. This model is a fine - tuned version of the pretrained model for Finnish ASR.

Intended uses & limitations

How to use

Check the run-finnish-asr-models.ipynb notebook in this repository for a detailed example of how to use this model.

Limitations and bias

• Audio length: This model was fine - tuned with audio samples of a maximum length of 20 seconds, so it likely works best for relatively short audios of similar length. However, you can try it with longer audios and see the performance. If you encounter out - of - memory errors with very long audio files, you can use the audio chunking method introduced in this blog post.
• Data domain: A large portion of the fine - tuning data was from the Finnish Parliament dataset, so the model may not generalize well to very different domains such as common daily spoken Finnish with dialects.
• Gender bias: The datasets' audios tend to be dominated by adult males, so the model may not perform as well for children's and women's speeches.
• Language model: The Finnish KenLM language model used in the decoding phase was trained with text data from audio transcriptions and a subset of Finnish Wikipedia. Thus, it may not generalize to very different language types, such as spoken daily language with dialects. It may be beneficial to train your own KenLM language model for your specific domain.

Training data

This model was fine - tuned with 276.7 hours of Finnish transcribed speech data from the following datasets:

The datasets were filtered to include audio samples with a maximum length of 20 seconds.

Training procedure

This model was trained on a Tesla V100 GPU, sponsored by Hugging Face & OVHcloud. The training script was provided by Hugging Face and is available here. We only modified its data loading for our custom datasets.

For the KenLM language model training, we followed the blog post tutorial provided by Hugging Face. The training data for the 5 - gram KenLM were text transcriptions of the audio training data and 100k random samples of the cleaned Finnish Wikipedia (August 2021) dataset.

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 1e - 04
• train_batch_size: 64
• eval_batch_size: 64
• seed: 42
• optimizer: 8 - bit Adam with betas=(0.9, 0.999) and epsilon = 1e - 08
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 500
• num_epochs: 10
• mixed_precision_training: Native AMP

The pretrained facebook/wav2vec2-base-fi-voxpopuli-v2 model was initialized with the following hyperparameters:

• attention_dropout: 0.094
• hidden_dropout: 0.047
• feat_proj_dropout: 0.04
• mask_time_prob: 0.082
• layerdrop: 0.041
• activation_dropout: 0.055
• ctc_loss_reduction: "mean"

Training results

Framework versions

• Transformers 4.19.1
• Pytorch 1.11.0+cu102
• Datasets 2.2.1
• Tokenizers 0.11.0

Evaluation results

Evaluation was done with the Common Voice 7.0 Finnish test split, Common Voice 9.0 Finnish test split, and the FLEURS ASR Finnish test split.

Note that the training data of this model includes the training splits of Common Voice 9.0, while most of our previous models include Common Voice 7.0. So, we ran tests for both versions. However, Common Voice doesn't seem to fully preserve the test split between dataset versions, so there may be some overlap between the training and test splits of different versions. Thus, the test result comparisons between models trained with different Common Voice versions are not entirely accurate but still meaningful.

Common Voice 7.0 testing

To evaluate this model, run the eval.py script in this repository:

python3 eval.py --model_id Finnish-NLP/wav2vec2-base-fi-voxpopuli-v2-finetuned --dataset mozilla-foundation/common_voice_7_0 --config fi --split test

This model (the first row of the table) achieves the following WER (Word Error Rate) and CER (Character Error Rate) results compared to our other models and their parameter counts:

Common Voice 9.0 testing

To evaluate this model, run the eval.py script in this repository:

python3 eval.py --model_id Finnish-NLP/wav2vec2-base-fi-voxpopuli-v2-finetuned --dataset mozilla-foundation/common_voice_9_0 --config fi --split test

This model (the first row of the table) achieves the following WER (Word Error Rate) and CER (Character Error Rate) results compared to our other models and their parameter counts:

FLEURS ASR testing

To evaluate this model, run the eval.py script in this repository:

python3 eval.py --model_id Finnish-NLP/wav2vec2-base-fi-voxpopuli-v2-finetuned --dataset google/fleurs --config fi_fi --split test

This model (the first row of the table) achieves the following WER (Word Error Rate) and CER (Character Error Rate) results compared to our other models and their parameter counts:

📄 License

This project is licensed under the Apache - 2.0 license.