Whisper Small Egyptian Arabic

🚀 Whisper Small - Fine-tuned for Egyptian Arabic ASR

This repository offers a fine - tuned openai/whisper-small model for Automatic Speech Recognition (ASR) focused on the Egyptian Arabic dialect, enhancing speech - to - text capabilities for this specific language variant.

🚀 Quick Start

You can use this model directly with the transformers library pipeline for automatic speech recognition. Ensure you have transformers and torch installed (pip install transformers torch).

from transformers import pipeline
import torch

# Ensure you have ffmpeg installed for audio processing
# pip install -U ffmpeg-python # or install via system package manager

device = "cuda:0" if torch.cuda.is_available() else "cpu"

# Replace "your-username/whisper-small-egyptian-arabic" with the actual model ID on the Hub
pipe = pipeline(
    "automatic-speech-recognition",
    model="your-username/whisper-small-egyptian-arabic", # <<< Replace this
    device=device
)

# Load your audio file (requires ffmpeg)
# For local files:
audio_file = "/path/to/your/egyptian_arabic_audio.wav"
result = pipe(audio_file, chunk_length_s=30, batch_size=8) # Adjust batch_size based on GPU memory

# For datasets library audio:
# from datasets import load_dataset
# ds = load_dataset("MAdel121/arabic-egy-cleaned", "default", split="test") # Example
# sample = ds[0]["audio"]
# result = pipe(sample.copy()) # Pass a copy to avoid modifying original

print(result["text"])

# --- Using AutoModelForSpeechSeq2Seq ---
from transformers import WhisperProcessor, WhisperForConditionalGeneration
import torchaudio

# Load the processor and model (replace with your model ID)
model_id = "your-username/whisper-small-egyptian-arabic" # <<< Replace this with your dataset file on hugging face
processor = WhisperProcessor.from_pretrained(model_id)
model = WhisperForConditionalGeneration.from_pretrained(model_id).to(device)

# Load and preprocess audio
waveform, sample_rate = torchaudio.load(audio_file)
if sample_rate != processor.feature_extractor.sampling_rate:
    resampler = torchaudio.transforms.Resample(sample_rate, processor.feature_extractor.sampling_rate)
    waveform = resampler(waveform)

input_features = processor(waveform.squeeze().numpy(), sampling_rate=processor.feature_extractor.sampling_rate, return_tensors="pt").input_features.to(device)

# Generate transcription
# Set forced_decoder_ids for Arabic transcription
forced_decoder_ids = processor.get_decoder_prompt_ids(language="ar", task="transcribe")
predicted_ids = model.generate(input_features, forced_decoder_ids=forced_decoder_ids)

transcription = processor.batch_decode(predicted_ids, skip_special_tokens=True)
print(transcription[0])

⚠️ Important Note

The original checkpoint was saved using SpeechBrain. This README assumes the model has been converted to the standard Hugging Face Transformers format for hosting and usage with the pipeline or AutoModel classes. If you are using the original .ckpt file, refer to the project's main README.md and the infer_whisper_local.py script for loading instructions.

✨ Features

• Dialect - Specific Focus: Tailored for the Egyptian Arabic dialect, providing more accurate speech - to - text results for this region.
• Fine - Tuned Model: Based on the openai/whisper-small model and fine - tuned using the SpeechBrain toolkit.

📦 Installation

Ensure you have transformers and torch installed. You can install them using the following command:

pip install transformers torch

Also, make sure to have ffmpeg installed for audio processing. You can install it via pip install -U ffmpeg-python or through your system package manager.

📚 Documentation

Model Description

Intended Uses & Limitations

This model is intended for transcribing speech in the Egyptian Arabic dialect.

Limitations:

• Performance may degrade significantly on other Arabic dialects.
• Performance on noisy audio may vary, as only specific augmentations (DropChunk, DropFreq, DropBitResolution) were used during training.
• The model might perform less effectively on highly specialized domains or topics not present in the fine - tuning dataset.

Training Data

The model was fine - tuned on the MAdel121/arabic-egy-cleaned dataset available on the Hugging Face Hub. This dataset contains cleaned audio samples and corresponding transcriptions in Egyptian Arabic.

Training Procedure

• Framework: SpeechBrain (speechbrain==1.0.3) with Hugging Face Transformers (transformers==4.51.3) and Accelerate (accelerate==0.25.0).
• Base Model: openai/whisper-small
• Dataset: MAdel121/arabic-egy-cleaned
• Epochs: 10
• Optimizer: AdamW (lr = 1e-5, weight_decay = 0.05)
• LR Scheduler: NewBob (improvement_threshold = 0.0025, annealing_factor = 0.9, patient = 0)
• Warmup Steps: 1000
• Batch Size: 8 (fixed, no dynamic batching)
• Gradient Accumulation: 2 steps (effective batch size: 16)
• Gradient Clipping: Max norm 5.0
• Mixed Precision: Not explicitly mentioned, assumed FP32 or handled by Accelerate/Trainer.
• Augmentation: Enabled (augment_prob_master = 0.5, min_augmentations = 1, max_augmentations = 3) with the following techniques applied randomly from the pool:
  • DropChunk (length: 1600 - 4800 samples, count: 1 - 5)
  • DropFreq (count: 1 - 3)
  • DropBitResolution
• Training Environment: Modal Labs (gpu = A100 - 40GB)

Evaluation Results

The model was evaluated on the test split of the MAdel121/arabic-egy-cleaned dataset.

WER (Word Error Rate) and CER (Character Error Rate) are reported. Lower is better.

Validation metrics at the end of training (Epoch 10):

• Validation WER: 22.79%
• Validation CER: 16.76%

📄 License

This model is released under the MIT license.

🔧 Technical Details

The model is a fine - tuned version of openai/whisper-small using the SpeechBrain toolkit. The fine - tuning process involved specific hyperparameters and data augmentations to adapt the model to the Egyptian Arabic dialect. The evaluation on the test split of the MAdel121/arabic-egy-cleaned dataset provides insights into its performance in terms of WER and CER.

📄 Citation

If you use this model, please consider citing the original Whisper paper and the dataset used:

@article{radford2023robust,
  title={Robust Speech Recognition via Large-Scale Weak Supervision},
  author={Radford, Alec and Kim, Jong Wook and Xu, Tao and Brockman, Greg and McLeavey, Christine and Sutskever, Ilya},
  journal={arXiv preprint arXiv:2212.04356},
  year={2023}
}

@misc{adel_mohamed_2024_12860997,
  author       = {Adel Mohamed},
  title        = {MAdel121/arabic-egy-cleaned},
  month        = jun,
  year         = 2024,
  publisher    = {Zenodo},
  doi          = {10.5281/zenodo.12860997},
  url          = {https://doi.org/10.5281/zenodo.12860997}
}

@misc{speechbrain,
  title={{SpeechBrain}: A General-Purpose Speech Toolkit},
  author={Mirco Ravanelli and Titouan Parcollet and Peter Plantinga and Aku Rouhe and Samuele Cornell and Loren Lugosch and Cem Subakan and Nauman Dawalatabad and Abdelwahab Heba and Jianyuan Zhong and Ju-Chieh Chou and Sung-Lin Yeh and Szu-Wei Fu and Chien-Feng Liao and Elena Rastorgueva and François Grondin and William Aris and Hwidong Na and Yan Gao and Renato De Mori and Yoshua Bengio},
  year={2021},
  eprint={2106.04624},
  archivePrefix={arXiv},
  primaryClass={eess.AS},
  note={arXiv:2106.04624}
}

Model Card Authors

[Your Name/Organization Here]

(Based on training run ceeu3g6c)