đ Wav2Vec2-Large for Keyword Spotting
This project presents a model for keyword spotting, leveraging the power of Wav2Vec2. It addresses the need for accurate and efficient keyword detection in speech audio, offering a reliable solution for various speech - related applications.
đ Quick Start
You can use the model via the Audio Classification pipeline:
from datasets import load_dataset
from transformers import pipeline
dataset = load_dataset("anton-l/superb_demo", "ks", split="test")
classifier = pipeline("audio-classification", model="superb/wav2vec2-large-superb-ks")
labels = classifier(dataset[0]["file"], top_k=5)
Or use the model directly:
import torch
from datasets import load_dataset
from transformers import Wav2Vec2ForSequenceClassification, Wav2Vec2FeatureExtractor
from torchaudio.sox_effects import apply_effects_file
effects = [["channels", "1"], ["rate", "16000"], ["gain", "-3.0"]]
def map_to_array(example):
speech, _ = apply_effects_file(example["file"], effects)
example["speech"] = speech.squeeze(0).numpy()
return example
dataset = load_dataset("anton-l/superb_demo", "ks", split="test")
dataset = dataset.map(map_to_array)
model = Wav2Vec2ForSequenceClassification.from_pretrained("superb/wav2vec2-large-superb-ks")
feature_extractor = Wav2Vec2FeatureExtractor.from_pretrained("superb/wav2vec2-large-superb-ks")
inputs = feature_extractor(dataset[:4]["speech"], sampling_rate=16000, padding=True, return_tensors="pt")
logits = model(**inputs).logits
predicted_ids = torch.argmax(logits, dim=-1)
labels = [model.config.id2label[_id] for _id in predicted_ids.tolist()]
⨠Features
đ Documentation
Model description
This is a ported version of
S3PRL's Wav2Vec2 for the SUPERB Keyword Spotting task.
The base model is wav2vec2-large-lv60, which is pretrained on 16kHz
sampled speech audio. When using the model make sure that your speech input is also sampled at 16Khz.
For more information refer to SUPERB: Speech processing Universal PERformance Benchmark
Task and dataset description
Keyword Spotting (KS) detects preregistered keywords by classifying utterances into a predefined set of
words. The task is usually performed on - device for the fast response time. Thus, accuracy, model size, and
inference time are all crucial. SUPERB uses the widely used
Speech Commands dataset v1.0 for the task.
The dataset consists of ten classes of keywords, a class for silence, and an unknown class to include the
false positive.
For the original model's training and evaluation instructions refer to the
S3PRL downstream task README.
đģ Usage Examples
Basic Usage
You can use the model via the Audio Classification pipeline:
from datasets import load_dataset
from transformers import pipeline
dataset = load_dataset("anton-l/superb_demo", "ks", split="test")
classifier = pipeline("audio-classification", model="superb/wav2vec2-large-superb-ks")
labels = classifier(dataset[0]["file"], top_k=5)
Advanced Usage
Use the model directly:
import torch
from datasets import load_dataset
from transformers import Wav2Vec2ForSequenceClassification, Wav2Vec2FeatureExtractor
from torchaudio.sox_effects import apply_effects_file
effects = [["channels", "1"], ["rate", "16000"], ["gain", "-3.0"]]
def map_to_array(example):
speech, _ = apply_effects_file(example["file"], effects)
example["speech"] = speech.squeeze(0).numpy()
return example
dataset = load_dataset("anton-l/superb_demo", "ks", split="test")
dataset = dataset.map(map_to_array)
model = Wav2Vec2ForSequenceClassification.from_pretrained("superb/wav2vec2-large-superb-ks")
feature_extractor = Wav2Vec2FeatureExtractor.from_pretrained("superb/wav2vec2-large-superb-ks")
inputs = feature_extractor(dataset[:4]["speech"], sampling_rate=16000, padding=True, return_tensors="pt")
logits = model(**inputs).logits
predicted_ids = torch.argmax(logits, dim=-1)
labels = [model.config.id2label[_id] for _id in predicted_ids.tolist()]
đ§ Technical Details
Eval results
The evaluation metric is accuracy.
|
s3prl |
transformers |
test |
0.9666 |
N/A |
BibTeX entry and citation info
@article{yang2021superb,
title={SUPERB: Speech processing Universal PERformance Benchmark},
author={Yang, Shu - wen and Chi, Po - Han and Chuang, Yung - Sung and Lai, Cheng - I Jeff and Lakhotia, Kushal and Lin, Yist Y and Liu, Andy T and Shi, Jiatong and Chang, Xuankai and Lin, Guan - Ting and others},
journal={arXiv preprint arXiv:2105.01051},
year={2021}
}
đ License
This project is licensed under the apache - 2.0
license.
đĻ Additional Information
Property |
Details |
Model Type |
A ported version of S3PRL's Wav2Vec2 for the SUPERB Keyword Spotting task |
Training Data |
Speech Commands dataset v1.0 |
â ī¸ Important Note
When using the model, make sure that your speech input is sampled at 16Khz as the base model is pretrained on 16kHz sampled speech audio.
đĄ Usage Tip
You can refer to the S3PRL downstream task README for the original model's training and evaluation instructions.