Ja Cascaded S2t Translation

🚀 Cascaded Japanese Speech2Text Translation

This project provides a pipeline for speech-to-text translation from Japanese speech to any target language text based on the cascaded approach, which combines ASR and translation, offering high - accuracy results.

🚀 Quick Start

The Cascaded Japanese Speech2Text Translation pipeline is designed to translate Japanese speech into text in various target languages. It first performs ASR on Japanese speech to get Japanese text and then translates it to the desired language.

✨ Features

• Cascaded Approach: Combines ASR and translation for high - accuracy results.
• Multiple Target Languages: Can translate to any language that the NLLB model is trained on.
• Model Flexibility: Allows the use of different NLLB models by setting model_translation.

📦 Installation

The installation is mainly about using the transformers library. You can install it via pip:

pip install transformers

💻 Usage Examples

Basic Usage

Here is an example to translate Japanese speech into English text: First, download a sample speech:

wget https://huggingface.co/datasets/japanese-asr/ja_asr.jsut_basic5000/resolve/main/sample.flac -O sample_ja.flac

Then, run the pipeline as below:

from transformers import pipeline

# load model
pipe = pipeline(
    model="japanese-asr/ja-cascaded-s2t-translation",
    model_kwargs={"attn_implementation": "sdpa"},
    model_translation="facebook/nllb-200-distilled-600M",
    tgt_lang="eng_Latn",
    chunk_length_s=15,
    trust_remote_code=True,
)

# translate
output = pipe("./sample_ja.flac")

Advanced Usage

Other NLLB models can be used by setting model_translation such as following:

• facebook/nllb-200-3.3B
• facebook/nllb-200-distilled-600M
• facebook/nllb-200-distilled-1.3B
• facebook/nllb-200-1.3B

📚 Documentation

Benchmark

The following table shows WER computed over the reference and predicted translation for translating Japanese speech to English text task (subsets of CoVoST2 and Fleurs) with different size of NLLB along with OpenAI Whisper models.

See https://github.com/kotoba-tech/kotoba-whisper for the evaluation detail.

Inference Speed

Due to the nature of cascaded approach, the pipeline has additional complexity compared to the single end2end OpenAI whisper models for the sake of high accuracy. The following table shows the mean inference time in second averaged over 10 trials on audio sample with different durations.

📄 License

This project is licensed under the Apache - 2.0 license.