Whisper Small Korean Pronunciation Scorer Sampledata

🚀 微調版Whisper發音評分器

該模型用於評估韓語語音的發音質量，基於openai/whisper-small模型，使用韓國AI-Hub（https://www.aihub.or.kr/）的外國人韓語發音評估數據集進行微調。

🚀 快速開始

本發音評分器接收音頻輸入及其對應的文本轉錄，並提供1到5分的韓語發音評分。它利用Whisper模型的編碼器 - 解碼器架構提取語音特徵，並使用一個額外的線性層來預測發音分數。

✨ 主要特性

• 專門針對韓語語音進行發音質量評估。
• 基於openai/whisper-small模型微調，利用其編碼器 - 解碼器架構提取特徵。
• 提供1到5分的發音評分。

📦 安裝指南

使用此模型，需按以下步驟操作：

1. 安裝所需庫
2. 加載模型和處理器
3. 準備音頻文件和文本轉錄
4. 預測發音分數

💻 使用示例

基礎用法

import torch
import torchaudio
from transformers import WhisperProcessor, WhisperForConditionalGeneration
import torch.nn as nn

class WhisperPronunciationScorer(nn.Module):
    def __init__(self, pretrained_model):
        super().__init__()
        self.whisper = pretrained_model
        self.score_head = nn.Linear(self.whisper.config.d_model, 1)

    def forward(self, input_features, labels=None):
        outputs = self.whisper(input_features, labels=labels, output_hidden_states=True)
        last_hidden_state = outputs.decoder_hidden_states[-1]
        scores = self.score_head(last_hidden_state.mean(dim=1)).squeeze()
        return scores

def load_model(model_path, device):
    model_name = "openai/whisper-small"
    processor = WhisperProcessor.from_pretrained(model_name)
    pretrained_model = WhisperForConditionalGeneration.from_pretrained(model_name)
    model = WhisperPronunciationScorer(pretrained_model).to(device)
    model.load_state_dict(torch.load(model_path, map_location=device))
    model.eval()
    return model, processor

def predict_pronunciation_score(model, processor, audio_path, transcript, device):
    # Load and preprocess audio
    audio, sr = torchaudio.load(audio_path)
    if sr != 16000:
        audio = torchaudio.functional.resample(audio, sr, 16000)
    input_features = processor(audio.squeeze().numpy(), sampling_rate=16000, return_tensors="pt").input_features.to(device)
    
    # Prepare transcript
    labels = processor(text=transcript, return_tensors="pt").input_ids.to(device)
    
    # Predict score
    with torch.no_grad():
        score = model(input_features, labels)
    return score.item()

# Load model
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model_path = "path/to/your/model.pth"
model, processor = load_model(model_path, device)

# Run prediction
audio_path = "path/to/your/audio.wav"
transcript = "안녕하세요"
score = predict_pronunciation_score(model, processor, audio_path, transcript, device)
print(f"Predicted pronunciation score: {score:.2f}")

高級用法

上述代碼已經包含了完整的使用流程，可根據實際需求對代碼進行修改，例如調整模型路徑、音頻文件路徑和文本轉錄內容等。

📄 許可證

本項目採用Apache-2.0許可證。

📚 詳細文檔