đ Kotoba-Whisper-v2.1
Kotoba-Whisper-v2.1 is a Japanese Automatic Speech Recognition (ASR) model. It builds upon kotoba-tech/kotoba-whisper-v2.0 and integrates additional post - processing stacks as a pipeline
. New features include adding punctuation with punctuators. These libraries are merged into Kotoba-Whisper-v2.1 via the pipeline and will be seamlessly applied to the predicted transcription from kotoba-tech/kotoba-whisper-v2.0. The pipeline was developed through the collaboration between Asahi Ushio and Kotoba Technologies.
đ Quick Start
Kotoba-Whisper-v2.1 is supported in the Hugging Face đ¤ Transformers library from version 4.39 onwards. To run the model, first install the latest version of Transformers.
pip install --upgrade pip
pip install --upgrade transformers accelerate torchaudio
pip install stable-ts==2.16.0
pip install punctuators==0.0.5
⨠Features
- Post - processing Integration: Integrates additional post - processing stacks as a pipeline, including punctuation addition.
- Multiple Dataset Support: Supports multiple Japanese speech datasets for evaluation and testing.
đĻ Installation
To install the necessary dependencies for running the model, use the following commands:
pip install --upgrade pip
pip install --upgrade transformers accelerate torchaudio
pip install stable-ts==2.16.0
pip install punctuators==0.0.5
đģ Usage Examples
Basic Usage
The model can be used with the pipeline
class to transcribe audio files as follows:
import torch
from transformers import pipeline
from datasets import load_dataset
model_id = "kotoba-tech/kotoba-whisper-v2.1"
torch_dtype = torch.float16 if torch.cuda.is_available() else torch.float32
device = "cuda:0" if torch.cuda.is_available() else "cpu"
model_kwargs = {"attn_implementation": "sdpa"} if torch.cuda.is_available() else {}
generate_kwargs = {"language": "ja", "task": "transcribe"}
pipe = pipeline(
model=model_id,
torch_dtype=torch_dtype,
device=device,
model_kwargs=model_kwargs,
batch_size=16,
trust_remote_code=True,
punctuator=True
)
dataset = load_dataset("japanese-asr/ja_asr.reazonspeech_test", split="test")
sample = dataset[0]["audio"]
result = pipe(sample, chunk_length_s=15, return_timestamps=True, generate_kwargs=generate_kwargs)
print(result)
Advanced Usage
Transcribe a Local Audio File
To transcribe a local audio file, simply pass the path to your audio file when you call the pipeline:
- result = pipe(sample, return_timestamps=True, generate_kwargs=generate_kwargs)
+ result = pipe("audio.mp3", return_timestamps=True, generate_kwargs=generate_kwargs)
Deactivate Punctuator
To deactivate the punctuator:
- punctuator=True,
+ punctuator=False,
Use Flash Attention 2
We recommend using Flash - Attention 2 if your GPU allows for it. To do so, you first need to install [Flash Attention](https://github.com/Dao - AILab/flash - attention):
pip install flash-attn --no-build-isolation
Then pass attn_implementation="flash_attention_2"
to from_pretrained
:
- model_kwargs = {"attn_implementation": "sdpa"} if torch.cuda.is_available() else {}
+ model_kwargs = {"attn_implementation": "flash_attention_2"} if torch.cuda.is_available() else {}
đ Documentation
Model Evaluation
The following table presents the raw CER (unlike usual CER where the punctuations are removed before computing the metrics, see the evaluation script here):
Regarding the normalized CER, since those updates from v2.1 will be removed by the normalization, kotoba-tech/kotoba-whisper-v2.1 marks the same CER values as kotoba-tech/kotoba-whisper-v2.0.
Latency
Please refer to the section of the latency in the kotoba-whisper-v1.1 here.
đ License
This project is licensed under the Apache 2.0 license.
đ§ Technical Details
The model is based on the Whisper architecture and is fine - tuned on Japanese speech datasets. It integrates post - processing libraries for punctuation addition and supports Flash Attention 2 for improved performance on compatible GPUs.
Acknowledgements