Whisper Base Vi

🚀 Whisper Base Vi V1.1: Whisper Base for Vietnamese Fine-Tuned by Nam Phung

This model is a fine - tuned version of the [openai/whisper - base](https://huggingface.co/openai/whisper - base) model. It is optimized for Vietnamese automatic speech recognition, aiming to enhance transcription accuracy and robustness, especially in real - world scenarios.

✨ Features

• High - Quality Fine - Tuning: Fine - tuned on over 100 hours of high - quality Vietnamese speech data to improve transcription accuracy for Vietnamese ASR tasks.
• Multilingual Foundation: Based on a model trained on over 680,000 hours of labeled audio data in multiple languages, with a focus on Vietnamese after fine - tuning.
• Efficient Processing: Works with the WhisperProcessor to pre - process audio inputs into log - Mel spectrograms and decode them into text.

📦 Installation

To use the fine - tuned model, you need to install the required dependencies:

# Install required libraries
!pip install transformers torch librosa soundfile --quiet

# Import necessary libraries
import torch
import librosa
import soundfile as sf
from transformers import AutoModelForSpeechSeq2Seq, AutoProcessor, pipeline

print("Environment setup completed!")

💻 Usage Examples

Basic Usage

You can use the model by following these steps:

import torch
import librosa
from transformers import AutoProcessor, AutoModelForSpeechSeq2Seq

device = "cuda" if torch.cuda.is_available() else "cpu"
print(f"Using device: {device}")

# Load processor and model
model_id = "namphungdn134/whisper - base - vi"
print(f"Loading model from: {model_id}")
processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForSpeechSeq2Seq.from_pretrained(model_id).to(device)

# config language and task
forced_decoder_ids = processor.get_decoder_prompt_ids(language="vi", task="transcribe")
model.config.forced_decoder_ids = forced_decoder_ids
print(f"Forced decoder IDs for Vietnamese: {forced_decoder_ids}")

# Preprocess
audio_path = "example.wav"  
print(f"Loading audio from: {audio_path}")
audio, sr = librosa.load(audio_path, sr = 16000)  
input_features = processor(audio, sampling_rate = 16000, return_tensors="pt").input_features.to(device)
print(f"Input features shape: {input_features.shape}")

# Generate
print("Generating transcription...")
with torch.no_grad():
    predicted_ids = model.generate(input_features, forced_decoder_ids = forced_decoder_ids)

transcription = processor.batch_decode(predicted_ids, skip_special_tokens = True)[0]
print("📝 Transcription:", transcription)

# Debug: Print token to check
print("Predicted IDs:", predicted_ids[0].tolist())

Or you can go to: https://github.com/namphung134/np - asr - vietnamese

📚 Documentation

Model Description

The Whisper Base model is a transformer - based sequence - to - sequence model for automatic speech recognition and translation tasks. It has been trained on a large amount of multilingual labeled audio data. The fine - tuned version focuses on the Vietnamese language, aiming to improve transcription accuracy and handle local dialects better.

Dataset

• Total Duration: More than 100 hours of high - quality Vietnamese speech data.
• Sources: Public Vietnamese datasets.
• Format: 16kHz WAV files with corresponding text transcripts.
• Preprocessing: Audio was normalized and segmented. Transcripts were cleaned and tokenized.

Fine - tuning Results

• Word Error Rate (WER): 16.9148. Evaluation was performed on a held - out test set with diverse regional accents and speaking styles.

🔧 Technical Details

This model is a fine - tuned version of [openai/whisper - base](https://huggingface.co/openai/whisper - base) on Vietnamese speech data. It uses the WhisperProcessor to convert audio inputs into a suitable format for the model and then decodes the outputs into text.

⚠️ Important Note

• This model is specifically fine - tuned for the Vietnamese language and may not perform well on other languages.
• It struggles with overlapping speech or noisy background.
• Performance may drop with strong dialectal variations not well represented in the training data.

📄 License

This model is licensed under the MIT License.

📚 Citation

If you use this model in your research or application, please cite the original Whisper model and this fine - tuning work as follows:

@article{Whisper2021,
  title={Whisper: A Multilingual Speech Recognition Model},
  author={OpenAI},
  year={2021},
  journal={arXiv:2202.12064},
  url={https://arxiv.org/abs/2202.12064}
}

@misc{title={Whisper Base Vi V1.1 - Nam Phung},
  author={Nam Phùng},
  organization={DUT},
  year={2025},
  url={https://huggingface.co/namphungdn134/whisper - base - vi}
}

📬 Contact

For questions, collaborations, or suggestions, feel free to reach out via [namphungdn134@gmail.com].