Phi 4 Multimodal Instruct Ko Asr

🚀 Phi-4-multimodal-instruct-ko-asr

This model is fine-tuned for Korean automatic speech recognition and translation, achieving high performance on multiple datasets.

🚀 Quick Start

This model is fine-tuned from microsoft/Phi-4-multimodal-instruct on Bingsu/zeroth-korean, google/flerus in 5 epochs.

This model is trained 960 steps on datasets for Korean Audio Speech Recognition on H100.

After that, we continue training with CoVoST2 Dataset / CoVoST2-Ko for AST.

AST Finetuned model is Here : Phi-4-multimodal-instruct-ko-speech

📚 Documentation

Model Information

Model Index

• Name: Phi-4-multimodal-instruct-ko-asr
• Results:
  • Task: Automatic Speech Recognition
  • Dataset: Bingsu/zeroth_korean (zeroth-korean-test)
  • Metrics:
    • BLEU: 94.837
    • CER: 1.316
    • WER: 2.951
  • Task: Automatic Speech Recognition
  • Dataset: google/flerus (flerus-ko-test)
  • Metrics:
    • BLEU: 67.659
    • CER: 7.951
    • WER: 18.313

🔧 Technical Details

This model is fine-tuned from microsoft/Phi-4-multimodal-instruct on Korean datasets for automatic speech recognition and translation. It is trained for 5 epochs on Bingsu/zeroth-korean and google/flerus, and then further trained on CoVoST2 Dataset / CoVoST2-Ko for AST.

📄 Evaluation

Evaluation was done on the following datasets:

• ASR (Automatic Speech Recognition): Evaluated with CER (Character Error Rate) on zeroth-test set (457 samples).
• AST (Automatic Speech Translation): Evaluated with BLEU score on fleurs ko <-> en speech translation result (270 samples).

Script is retrieved from here.

Compared to Phi-4-mm-inst-zeroth-kor and Phi-4-multimodal-finetune-ko-speech, ASR is significantly improved.