Asr Streaming Conformer Gigaspeech

🚀 Conformer for GigaSpeech

This repository offers all the essential tools for automatic speech recognition using an end - to - end system pretrained on GigaSpeech (EN, XL split) within SpeechBrain. It's recommended to learn more about SpeechBrain for a better experience.

Model Information

Model Performance

The performance of the model in full context mode (no streaming) is as follows:

With streaming, the results with different chunk sizes on the test split are as follows:

(*: model was never explicitly trained with this setting)

⚠️ Important Note

When comparing configurations, keep in mind that the utterances of the test set are of a finite length. All WER values were evaluated using this script (until a better way to evaluate streaming WER in different conditions is integrated).

🚀 Quick Start

Install SpeechBrain

First, install SpeechBrain using the following command:

pip install speechbrain

💡 Usage Tip

It's recommended to read the tutorials and learn more about SpeechBrain.

Transcribing your own audio files (in English)

from speechbrain.inference.ASR import StreamingASR
from speechbrain.utils.dynamic_chunk_training import DynChunkTrainConfig
asr_model = StreamingASR.from_hparams("speechbrain/asr-streaming-conformer-gigaspeech")
asr_model.transcribe_file(
    "speechbrain/asr-streaming-conformer-librispeech/test-en.wav",
    # select a chunk size of ~960ms with 4 chunks of left context
    DynChunkTrainConfig(24, 4),
    # disable torchaudio streaming to allow fetching from HuggingFace
    # set this to True for your own files or streams to allow for streaming file decoding
    use_torchaudio_streaming=False,
)

The DynChunkTrainConfig values can be adjusted to balance latency, computational power, and transcription accuracy. Refer to the streaming WER table to choose a suitable value.

Inference on GPU

To perform inference on the GPU, add run_opts={"device":"cuda"} when calling the from_hparams method.

✨ Features

• End - to - End ASR: Provides an end - to - end automatic speech recognition solution pretrained on GigaSpeech.
• Streaming Support: Supports streaming ASR with Dynamic Chunk Training.
• Multi - Chunk Size Compatibility: Trained with support for different chunk sizes and full context, suitable for various scenarios.

📦 Installation

First, clone SpeechBrain:

git clone https://github.com/speechbrain/speechbrain/

Then, install it:

cd speechbrain
pip install -r requirements.txt
pip install -e .

💻 Usage Examples

Basic Usage

from speechbrain.inference.ASR import StreamingASR
from speechbrain.utils.dynamic_chunk_training import DynChunkTrainConfig
asr_model = StreamingASR.from_hparams("speechbrain/asr-streaming-conformer-gigaspeech")
asr_model.transcribe_file(
    "speechbrain/asr-streaming-conformer-librispeech/test-en.wav",
    DynChunkTrainConfig(24, 4),
    use_torchaudio_streaming=False
)

Advanced Usage

Transcribing from a live stream using ffmpeg

python3 asr.py http://as-hls-ww-live.akamaized.net/pool_904/live/ww/bbc_radio_fourfm/bbc_radio_fourfm.isml/bbc_radio_fourfm-audio%3d96000.norewind.m3u8 --model-source=speechbrain/asr-streaming-conformer-gigaspeech --device=cpu -v

Transcribing from a file

python3 asr.py some-english-speech.wav --model-source=speechbrain/asr-streaming-conformer-gigaspeech --device=cpu -v

Live ASR decoding from a browser using Gradio

from argparse import ArgumentParser
from dataclasses import dataclass
import logging

parser = ArgumentParser()
parser.add_argument("--model-source", required=True)
parser.add_argument("--device", default="cpu")
parser.add_argument("--ip", default="127.0.0.1")
parser.add_argument("--port", default=9431)
parser.add_argument("--chunk-size", default=24, type=int)
parser.add_argument("--left-context-chunks", default=4, type=int)
parser.add_argument("--num-threads", default=None, type=int)
parser.add_argument("--verbose", "-v", default=False, action="store_true")
args = parser.parse_args()

if args.verbose:
    logging.getLogger().setLevel(logging.INFO)

logging.info("Loading libraries")

from speechbrain.inference.ASR import StreamingASR, ASRStreamingContext
from speechbrain.utils.dynamic_chunk_training import DynChunkTrainConfig
import torch
import gradio as gr
import torchaudio
import numpy as np

device = args.device

if args.num_threads is not None:
    torch.set_num_threads(args.num_threads)

logging.info(f"Loading model from \"{args.model_source}\" onto device {device}")

asr = StreamingASR.from_hparams(args.model_source, run_opts={"device": device})
config = DynChunkTrainConfig(args.chunk_size, args.left_context_chunks)

@dataclass
class GradioStreamingContext:
    context: ASRStreamingContext
    chunk_size: int
    waveform_buffer: torch.Tensor
    decoded_text: str

def transcribe(stream, new_chunk):
    sr, y = new_chunk

    y = y.astype(np.float32)
    y = torch.tensor(y, dtype=torch.float32, device=device)
    y /= max(1, torch.max(torch.abs(y)).item())  # norm by max abs() within chunk & avoid NaN
    if len(y.shape) > 1:
        y = torch.mean(y, dim=1)  # downmix to mono

    # HACK: we are making poor use of the resampler across chunk boundaries
    # which may degrade accuracy.
    # NOTE: we should also absolutely avoid recreating a resampler every time
    resampler = torchaudio.transforms.Resample(orig_freq=sr, new_freq=asr.audio_normalizer.sample_rate).to(device)
    y = resampler(y)  # janky resample (probably to 16kHz)


    if stream is None:
        stream = GradioStreamingContext(
            context=asr.make_streaming_context(config),
            chunk_size=asr.get_chunk_size_frames(config),
            waveform_buffer=y,
            decoded_text="",
        )
    else:
        stream.waveform_buffer = torch.concat((stream.waveform_buffer, y))

    while stream.waveform_buffer.size(0) > stream.chunk_size:
        chunk = stream.waveform_buffer[:stream.chunk_size]
        stream.waveform_buffer = stream.waveform_buffer[stream.chunk_size:]

        # fake batch dim
        chunk = chunk.unsqueeze(0)

        # list of transcribed strings, of size 1 because the batch size is 1
        with torch.no_grad():
            transcribed = asr.transcribe_chunk(stream.context, chunk)
        stream.decoded_text += transcribed[0]

    return stream, stream.decoded_text

# NOTE: latency seems relatively high, which may be due to this:
# https://github.com/gradio-app/gradio/issues/6526

demo = gr.Interface(
    transcribe,
    ["state", gr.Audio(sources=["microphone"], streaming=True)],
    ["state", "text"],
    live=True,
)

demo.launch(server_name=args.ip, server_port=args.port)

📚 Documentation

Pipeline description

This ASR system is a Conformer model trained with the RNN - T loss (with an auxiliary CTC loss to stabilize training). The model operates with a unigram tokenizer. Architecture details are described in the training hyperparameters file.

Streaming support makes use of Dynamic Chunk Training. Chunked attention is used for the multi - head attention module, and an implementation of Dynamic Chunk Convolutions was used. The model was trained with support for different chunk sizes (and even full context), and so is suitable for various chunk sizes and offline transcription.

The system is trained with recordings sampled at 16kHz (single channel). The code will automatically normalize your audio (i.e., resampling + mono channel selection) when calling transcribe_file if needed.

Parallel Inference on a Batch

Currently, the high - level transcription interfaces do not support batched inference, but the low - level interfaces (i.e., encode_chunk) do. There are plans to provide efficient functionality for this in the future.

Training

The model was trained with SpeechBrain v1.0.2. To train it from scratch, follow these steps:

1. Clone SpeechBrain:

git clone https://github.com/speechbrain/speechbrain/

2. Install it:

cd speechbrain
pip install -r requirements.txt
pip install -e .

3. Follow the steps listed in the README for this recipe.

Limitations

The SpeechBrain team does not provide any warranty on the performance achieved by this model when used on other datasets.

🔧 Technical Details

The model was trained with support for different chunk sizes and full context, which makes it suitable for various chunk sizes and offline transcription. Dynamic Chunk Training is used for streaming support, and an implementation of Dynamic Chunk Convolutions was applied.

📄 License

This project is licensed under the apache - 2.0 license.

About SpeechBrain

• Website: https://speechbrain.github.io/
• Code: https://github.com/speechbrain/speechbrain/
• HuggingFace: https://huggingface.co/speechbrain/

Citing SpeechBrain

Please cite SpeechBrain if you use it for your research or business.

@misc{speechbrainV1,
  title={Open-Source Conversational AI with SpeechBrain 1.0},
  author={Mirco Ravanelli and Titouan Parcollet and Adel Moumen and Sylvain de Langen and Cem Subakan and Peter Plantinga and Yingzhi Wang and Pooneh Mousavi and Luca Della Libera and Artem Ploujnikov and Francesco Paissan and Davide Borra and Salah Zaiem and Zeyu Zhao and Shucong Zhang and Georgios Karakasidis and Sung-Lin Yeh and Pierre Champion and Aku Rouhe and Rudolf Braun and Florian Mai and Juan Zuluaga-Gomez and Seyed Mahed Mousavi and Andreas Nautsch and Xuechen Liu and Sangeet Sagar and Jarod Duret and Salima Mdhaffar and Gaelle Laperriere and Mickael Rouvier and Renato De Mori and Yannick Esteve},
  year={2024},
  eprint={2407.00463},
  archivePrefix={arXiv},
  primaryClass={cs.LG},
  url={https://arxiv.org/abs/2407.00463},
}
@misc{speechbrain,
  title={{SpeechBrain}: A General-Purpose Speech Toolkit},
  author={Mirco Ravanelli and Titouan Parcollet and Peter Plantinga and Aku Rouhe and Samuele Cornell and Loren Lugosch and Cem Subakan and Nauman Dawalatabad and Abdelwahab Heba and Jianyuan Zhong and Ju-Chieh Chou and Sung-Lin Yeh and Szu-Wei Fu and Chien-Feng Liao and Elena Rastorgueva and François Grondin and William Aris and Hwidong Na and Yan Gao and Renato De Mori and Yoshua Bengio},
  year={2021},
  eprint={2106.04624},
  archivePrefix={arXiv},
  primaryClass={eess.AS},
  note={arXiv:2106.04624}
}