Crisperwhisper

🚀 CrisperWhisper

CrisperWhisper is an advanced speech recognition model. It's a variant of OpenAI's Whisper, designed for fast, precise, and verbatim speech recognition with accurate word - level timestamps. It aims to transcribe every spoken word exactly as it is, including fillers, pauses, stutters and false starts.

📦 Model Information

📢 Additional Information

• See our collection for all our TTS model uploads.
• Learn to fine - tune TTS models - Read our Guide.
• Unsloth Dynamic 2.0 achieves superior accuracy & outperforms other leading quants.

✨ Features

• 🎯 Accurate Word - Level Timestamps: Provides precise timestamps, even around disfluencies and pauses, by utilizing an adjusted tokenizer and a custom attention loss during training.
• 📝 Verbatim Transcription: Transcribes every spoken word exactly as it is, including and differentiating fillers like "um" and "uh".
• 🔍 Filler Detection: Detects and accurately transcribes fillers.
• 🛡️ Hallucination Mitigation: Minimizes transcription hallucinations to enhance accuracy.

📈 Performance Overview

🔍 Qualitative Performance Overview

📊 Quantitative Performance Overview

Transcription Performance

CrisperWhisper significantly outperforms Whisper Large v3, especially on datasets that have a more verbatim transcription style in the ground truth, such as AMI and TED - LIUM.

Segmentation Performance

CrisperWhisper demonstrates superior performance segmentation performance. This performance gap is especially pronounced around disfluencies and pauses. The following table uses the metrics as defined in the paper. For this table we used a collar of 50ms. Heads for each Model were selected using the method described in the How section and the result attaining the highest F1 Score was choosen for each model using varying number of heads.

💻 Usage Examples

Basic Usage

First, install our custom transformers fork for the most accurate timestamps:

pip install git+https://github.com/nyrahealth/transformers.git@crisper_whisper

Advanced Usage

import os
import sys
import torch

from datasets import load_dataset
from transformers import AutoModelForSpeechSeq2Seq, AutoProcessor, pipeline

def adjust_pauses_for_hf_pipeline_output(pipeline_output, split_threshold=0.12):
    """
    Adjust pause timings by distributing pauses up to the threshold evenly between adjacent words.
    """

    adjusted_chunks = pipeline_output["chunks"].copy()

    for i in range(len(adjusted_chunks) - 1):
        current_chunk = adjusted_chunks[i]
        next_chunk = adjusted_chunks[i + 1]

        current_start, current_end = current_chunk["timestamp"]
        next_start, next_end = next_chunk["timestamp"]
        pause_duration = next_start - current_end

        if pause_duration > 0:
            if pause_duration > split_threshold:
                distribute = split_threshold / 2
            else:
                distribute = pause_duration / 2

            # Adjust current chunk end time
            adjusted_chunks[i]["timestamp"] = (current_start, current_end + distribute)

            # Adjust next chunk start time
            adjusted_chunks[i + 1]["timestamp"] = (next_start - distribute, next_end)
    pipeline_output["chunks"] = adjusted_chunks

    return pipeline_output


device = "cuda:0" if torch.cuda.is_available() else "cpu"
torch_dtype = torch.float16 if torch.cuda.is_available() else torch.float32

model_id = "nyrahealth/CrisperWhisper"

model = AutoModelForSpeechSeq2Seq.from_pretrained

🌟 Highlights

• 🏆 1st place on the OpenASR Leaderboard in verbatim datasets (TED, AMI)
• 🎓 Accepted at INTERSPEECH 2024.
• 📄 Paper Drop: Check out our paper for details and reasoning behind our tokenizer adjustment.
• ✨ New Feature: Not mentioned in the paper is a added AttentionLoss to further improve timestamp accuracy. By specifically adding a loss to train the attention scores used for the DTW alignment using timestamped data we significantly boosted the alignment performance.

📄 License

This project is licensed under the CC - BY - NC - 4.0 license.