Wav2vec2 Xls R 300m Cs Cv8

🚀 wav2vec2-xls-r-300m-cs-cv8

This model is a fine - tuned version of facebook/wav2vec2-xls-r-300m on the Common Voice 8.0 dataset. It is designed for automatic speech recognition, offering a practical solution for converting speech to text in Czech language.

✨ Features

• Fine - tuned Model: Based on facebook/wav2vec2-xls-r-300m, fine - tuned on the Common Voice 8.0 dataset.
• Multilingual Adaptability: Can be used for Czech speech recognition tasks.
• Evaluation Metrics: Achieved specific WER and CER results on different evaluation datasets.

📦 Installation

There is no specific installation steps provided in the original document. So this section is skipped.

💻 Usage Examples

Basic Usage

import torch
import torchaudio
from datasets import load_dataset
from transformers import Wav2Vec2ForCTC, Wav2Vec2Processor

test_dataset = load_dataset("mozilla-foundation/common_voice_8_0", "cs", split="test[:2%]")

processor = Wav2Vec2Processor.from_pretrained("comodoro/wav2vec2-xls-r-300m-cs-cv8")
model = Wav2Vec2ForCTC.from_pretrained("comodoro/wav2vec2-xls-r-300m-cs-cv8")

resampler = torchaudio.transforms.Resample(48_000, 16_000)

# Preprocessing the datasets.
# We need to read the aduio files as arrays
def speech_file_to_array_fn(batch):
	speech_array, sampling_rate = torchaudio.load(batch["path"])
	batch["speech"] = resampler(speech_array).squeeze().numpy()
	return batch

test_dataset = test_dataset.map(speech_file_to_array_fn)
inputs = processor(test_dataset[:2]["speech"], sampling_rate=16_000, return_tensors="pt", padding=True)

with torch.no_grad():
	logits = model(inputs.input_values, attention_mask=inputs.attention_mask).logits

predicted_ids = torch.argmax(logits, dim=-1)

print("Prediction:", processor.batch_decode(predicted_ids))
print("Reference:", test_dataset[:2]["sentence"])

📚 Documentation

Evaluation

The model can be evaluated using the attached eval.py script:

python eval.py --model_id comodoro/wav2vec2-xls-r-300m-cs-cv8 --dataset mozilla-foundation/common-voice_8_0 --split test --config cs

Training and evaluation data

The Common Voice 8.0 train and validation datasets were used for training.

Training procedure

Training hyperparameters

The following hyperparameters were used during first stage of training:

• learning_rate: 7e - 05
• train_batch_size: 32
• eval_batch_size: 8
• seed: 42
• gradient_accumulation_steps: 20
• total_train_batch_size: 640
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e - 08
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 500
• num_epochs: 150
• mixed_precision_training: Native AMP

The following hyperparameters were used during second stage of training:

• learning_rate: 0.001
• train_batch_size: 32
• eval_batch_size: 8
• seed: 42
• gradient_accumulation_steps: 20
• total_train_batch_size: 640
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e - 08
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 500
• num_epochs: 50
• mixed_precision_training: Native AMP

Training results

Further fine - tuning with slightly different architecture and higher learning rate:

Framework versions

• Transformers 4.16.0.dev0
• Pytorch 1.10.1+cu102
• Datasets 1.17.1.dev0
• Tokenizers 0.11.0

🔧 Technical Details

The model is a fine - tuned version of facebook/wav2vec2-xls-r-300m on the Common Voice 8.0 dataset. During training, different hyperparameters were used in two stages, and the training results were recorded in terms of training loss, validation loss, WER, and CER.

📄 License

This model is licensed under the Apache - 2.0 license.

📋 Model Information