Anime Whisper

🚀 Anime Whisper 🤗🎤📝

Anime Whisper is a Japanese speech recognition model specifically fine - tuned for the domain of anime - style acting voices, offering high - performance and unique features.

Anime Whisper is a Japanese speech recognition model specialized in the domain of anime - style acting voices, especially for Japanese. This model is fine - tuned from the [kotoba - whisper - v2.0](https://huggingface.co/kotoba - tech/kotoba - whisper - v2.0) base model using approximately 5,300 hours and 3.73 million files of anime - style voice and script datasets, such as Galgame_Speech_ASR_16kHz. Although it is specialized in the anime acting voice domain, it also demonstrates features and high performance not found in other models for other types of voices.

You can try the demo here: https://huggingface.co/spaces/litagin/anime - whisper - demo

🚀 Quick Start

✨ Features

Anime Whisper generally has the following tendencies compared to other models:

• Fewer hallucinations: It produces fewer hallucinations during the speech - to - text process.
• Faithful transcription: It can accurately transcribe non - linguistic utterances such as stutters, laughter, shouts, and breaths that are often skipped by other models.
• Appropriate punctuation: Punctuations like 。、!?… are added appropriately according to the rhythm and emotion of the speech, resulting in a natural - sounding script.
• High accuracy for anime voices: It shows particularly high accuracy for anime - style acting voices.
• Lightweight and fast: Based on [kotoba - whisper](https://huggingface.co/kotoba - tech/kotoba - whisper - v2.0) (a distilled model of [whisper - large - v3](https://huggingface.co/openai/whisper - large - v3)), it is lightweight and fast.
• NSFW voice transcription: It can transcribe NSFW voices in a proper style, which is almost impossible for other models.

📦 Installation

There is no specific installation content provided in the original document, so this section is skipped.

💻 Usage Examples

Basic Usage

import torch
from transformers import pipeline

generate_kwargs = {
    "language": "Japanese",
    "no_repeat_ngram_size": 0,
    "repetition_penalty": 1.0,
}
pipe = pipeline(
    "automatic - speech - recognition",
    model="litagin/anime - whisper",
    device="cuda",
    torch_dtype=torch.float16,
    chunk_length_s=30.0,
    batch_size=64,
)

audio_path = "test.wav"
result = pipe(audio_path, generate_kwargs=generate_kwargs)
print(result["text"])

• Multiple files inference: If you want to infer multiple files at once, simply pass a list of file paths to pipe.
• Suppressing hallucinations: If repeated hallucinations are noticeable, you can set no_repeat_ngram_size: int to around 5 - 10 or set repetition_penalty to a value greater than 1 to suppress them.

📚 Documentation

Evaluation 📊

Detailed evaluation reports, observation reports, and evaluation codes will be published on the [GitHub repository](https://github.com/litagin02/anime - whisper).

CER (Character Error Rate)

• Evaluation data: Evaluated on 5 personal novel games (approximately 75k files in total) that are in the same anime - style dialogue domain as the training data but not included in the training data.
• Generation parameters: Generated with the no_repeat_ngram_size = 5 parameter to suppress repeated hallucinations in OpenAI's Whisper series.
• CER calculation: CER is calculated based on the appropriately normalized results.

Bias and Other Issues 🚨

• Proper nouns: Proper nouns such as personal names in the visual novels of the training data are often transcribed in the Chinese characters used in the game.
• Specific words: Some specific words in the dataset may be transcribed differently from the norm (e.g., からだ → 身体 and other proper nouns).
• Normalization effects: Due to dataset normalization, the following rarely appear in the output:
  • Consecutive vowels or long vowels: ああああーーーー
  • Consecutive exclamation marks: こらーっ!!!! なにそれ!?!?!?!?
  • Consecutive ellipses: …… (usually only one … is output instead of the correct two …… in Japanese notation).
• Number, alphabet, and exclamation mark: Numbers, alphabets, and exclamation marks are transcribed in half - width characters.
• Ending punctuation: The ending 。 is almost always omitted.
• Vulgar language: Transcriptions of some vulgar language may contain censored characters like ○.

Examples 👀

This is a comparison of transcriptions of dialogue from a novel game that is not included in the training data (generated with no_repeat_ngram_size = 5 as well).

The results show that Anime Whisper generally performs as well as whisper - large - v3. The following are some examples highlighting the significant differences from other models, especially for non - linguistic utterances and emotional voices.

NSFW Examples 🫣

Please note that these examples contain adult - oriented expressions.

Training Procedures 📚

Detailed training procedures, hyperparameters, and training codes will be published on [GitHub](https://github.com/litagin02/anime - whisper) soon.

• Data split: The last tar file of all the data was reserved as test data, and the remaining 3,735,363 files were used for training.
• Initial training: First, freeze the Encoder from the base model and train only the Decoder for several epochs.
• Full - model training: Then, unfreeze the Encoder and train the entire model for several epochs.
• Model optimization: After stopping the training, an attempt was made to improve performance by taking the average (merging) of models from one point in time to another. Optuna was used to optimize the CER on the benchmark data, and the result was used as the final model.

Environment 🖥

• Hardware: The model was trained on an H100 NVL (VRAM 96GB) rented from vast.ai for less than 3 weeks through trial - and - error (initially, the base model was whisper - large - v3 - turbo, so this time is also included).
• Training time: The actual training time used for this model was approximately H100 NVL * 11.2 days. However, the performance on the test data in the latter part was probably poor due to over - fitting, so these models were not used in the final merge.

🔧 Technical Details

There is no specific technical details content provided in the original document, so this section is skipped.

📄 License

This project is licensed under the MIT license.