Ru Whisper Small

🚀 ru_whisper_small - Val123val

本模型是 openai/whisper-small 在 Sberdevices_golos_10h_crowd 數據集上進行微調後的版本，可作為自動語音識別（ASR）解決方案，尤其適用於俄語語音識別。

✨ 主要特性

• Whisper 是基於 Transformer 的編解碼器模型，也稱為序列到序列模型，在 680k 小時的標註語音數據上進行訓練，其中俄語數據僅有 5k 小時。
• ru_whisper_small 是在 Sberdevices_golos_10h_crowd 數據集上微調的版本，對於開發者來說，可能是一個很有用的 ASR 解決方案，特別是在俄語語音識別方面。如果針對特定業務任務進行微調，還可能展現出額外的能力。

📦 安裝指南

文檔未提供具體安裝步驟，可參考相關庫的官方文檔進行安裝，如 transformers、datasets 等。

💻 使用示例

基礎用法

from transformers import WhisperProcessor, WhisperForConditionalGeneration
from datasets import load_dataset

# 加載模型和處理器
processor = WhisperProcessor.from_pretrained("Val123val/ru_whisper_small")
model = WhisperForConditionalGeneration.from_pretrained("Val123val/ru_whisper_small")
model.config.forced_decoder_ids = None

# 加載數據集並讀取音頻文件
ds = load_dataset("bond005/sberdevices_golos_10h_crowd", split="validation", token=True)
sample = ds[0]["audio"]
input_features = processor(sample["array"], sampling_rate=sample["sampling_rate"], return_tensors="pt").input_features 

# 生成令牌 ID
predicted_ids = model.generate(input_features)
# 將令牌 ID 解碼為文本
transcription = processor.batch_decode(predicted_ids, skip_special_tokens=False)

transcription = processor.batch_decode(predicted_ids, skip_special_tokens=True)

長音頻轉錄

import torch
from transformers import pipeline
from datasets import load_dataset

device = "cuda:0" if torch.cuda.is_available() else "cpu"

pipe = pipeline(
  "automatic-speech-recognition",
  model="Val123val/ru_whisper_small",
  chunk_length_s=30,
  device=device,
)

ds = load_dataset("bond005/sberdevices_golos_10h_crowd", split="validation", token=True)
sample = ds[0]["audio"]

prediction = pipe(sample.copy(), batch_size=8)["text"]

# 也可以返回預測的時間戳
prediction = pipe(sample.copy(), batch_size=8, return_timestamps=True)["chunks"]

基於推測解碼的加速使用

import torch
from transformers import AutoModelForSpeechSeq2Seq, AutoProcessor
from transformers import pipeline

device = "cuda:0" if torch.cuda.is_available() else "cpu"
torch_dtype = torch.float16 if torch.cuda.is_available() else torch.float32

# 加載數據集
dataset = load_dataset("bond005/sberdevices_golos_10h_crowd", split="validation", token=True)

# 加載模型
model_id = "Val123val/ru_whisper_small"

model = AutoModelForSpeechSeq2Seq.from_pretrained(
    model_id,
    torch_dtype=torch_dtype,
    low_cpu_mem_usage=True,
    use_safetensors=True,
    attn_implementation="sdpa",
)
model.to(device)

processor = AutoProcessor.from_pretrained(model_id)

# 加載輔助模型
assistant_model_id = "openai/whisper-tiny"

assistant_model = AutoModelForSpeechSeq2Seq.from_pretrained(
    assistant_model_id,
    torch_dtype=torch_dtype,
    low_cpu_mem_usage=True,
    use_safetensors=True,
    attn_implementation="sdpa",
)

assistant_model.to(device);

# 創建管道
pipe = pipeline(
    "automatic-speech-recognition",
    model=model,
    tokenizer=processor.tokenizer,
    feature_extractor=processor.feature_extractor,
    max_new_tokens=128,
    chunk_length_s=15,
    batch_size=4,
    generate_kwargs={"assistant_model": assistant_model},
    torch_dtype=torch_dtype,
    device=device,
)

sample = dataset[0]["audio"]
result = pipe(sample)
print(result["text"])

📚 詳細文檔

訓練超參數

訓練過程中使用了以下超參數：

框架版本

• Transformers 4.36.2
• Pytorch 2.1.0+cu121
• Datasets 2.16.0
• Tokenizers 0.15.0

📄 許可證

本項目採用 Apache-2.0 許可證。