Sew D Tiny 100k Ft Ls100h

🚀 SEW-D-tiny

SEW-D-tiny是基于16kHz采样语音音频预训练的基础模型。该模型可用于自动语音识别、说话人识别、意图分类、情感识别等下游任务。使用时，请确保输入的语音也采样为16kHz。

🔍 模型信息

📚 相关链接

• SEW-D by ASAPP Research
• Performance-Efficiency Trade-offs in Unsupervised Pre-training for Speech Recognition

📖 论文信息

• 标题：Performance-Efficiency Trade-offs in Unsupervised Pre-training for Speech Recognition
• 作者：Felix Wu, Kwangyoun Kim, Jing Pan, Kyu Han, Kilian Q. Weinberger, Yoav Artzi
• 摘要：本文研究了自动语音识别（ASR）预训练模型中的性能 - 效率权衡问题。聚焦于wav2vec 2.0，本文提出了几种影响模型性能和效率的架构设计。综合所有观察结果，引入了SEW（Squeezed and Efficient Wav2vec），这是一种在各种训练设置下，在性能和效率方面都有显著改进的预训练模型架构。例如，在LibriSpeech的100h - 960h半监督设置下，与wav2vec 2.0相比，SEW的推理速度提高了1.9倍，单词错误率相对降低了13.5%。在相似的推理时间内，SEW在不同模型大小下将单词错误率降低了25 - 50%。
• 原始模型：https://github.com/asappresearch/sew#model-checkpoints

🚀 快速开始

💻 使用示例

基础用法

from transformers import Wav2Vec2Processor, SEWDForCTC
from datasets import load_dataset
import soundfile as sf
import torch
 
# load the model and preprocessor
processor = Wav2Vec2Processor.from_pretrained("asapp/sew-d-tiny-100k-ft-ls100h")
model = SEWDForCTC.from_pretrained("asapp/sew-d-tiny-100k-ft-ls100h")

# load the dummy dataset with speech samples
ds = load_dataset("patrickvonplaten/librispeech_asr_dummy", "clean", split="validation")
 
# preprocess
input_values = processor(ds[0]["audio"]["array"], return_tensors="pt").input_values  # Batch size 1

# retrieve logits
logits = model(input_values).logits
 
# take argmax and decode
predicted_ids = torch.argmax(logits, dim=-1)
transcription = processor.batch_decode(predicted_ids)

评估示例

from datasets import load_dataset
from transformers import SEWDForCTC, Wav2Vec2Processor
import torch
from jiwer import wer

librispeech_eval = load_dataset("librispeech_asr", "clean", split="test")

model = SEWDForCTC.from_pretrained("asapp/sew-d-tiny-100k-ft-ls100h").to("cuda")
processor = Wav2Vec2Processor.from_pretrained("asapp/sew-d-tiny-100k-ft-ls100h")

def map_to_pred(batch):
    input_values = processor(batch["audio"][0]["array"], sampling_rate=16000, 
                             return_tensors="pt", padding="longest").input_values
    with torch.no_grad():
        logits = model(input_values.to("cuda")).logits

    predicted_ids = torch.argmax(logits, dim=-1)
    transcription = processor.batch_decode(predicted_ids)
    batch["transcription"] = transcription
    return batch

result = librispeech_eval.map(map_to_pred, batched=True, batch_size=1, remove_columns=["audio"])

print("WER:", wer(result["text"], result["transcription"]))

📊 评估结果

📄 许可证

本项目采用Apache-2.0许可证。