Smart Turn V2

🚀 Smart Turn v2

Smart Turn v2 is an open‑source semantic Voice Activity Detection (VAD) model. It analyzes the raw waveform, not the transcript, to determine whether a speaker has finished their turn. Compared with v1, it offers the following improvements:

• Multilingual – Supports 14 languages (EN, FR, DE, ES, PT, ZH, JA, HI, IT, KO, NL, PL, RU, TR).
• 6 × smaller – Approximately 360 MB, compared to 2.3 GB in v1.
• 3 × faster – It takes about 12 ms to analyze 8 s of audio on an NVIDIA L40S.

🚀 Quick Start

from transformers import pipeline
import soundfile as sf

pipe = pipeline(
    "audio-classification",
    model="pipecat-ai/smart-turn-v2",
    feature_extractor="facebook/wav2vec2-base"
)

speech, sr = sf.read("user_utterance.wav")
if sr != 16_000:
    raise ValueError("Resample to 16 kHz")

result = pipe(speech, top_k=None)[0]
print(f"Completed turn? {result['label']}  Prob: {result['score']:.3f}")
# label == 'complete' → user has finished speaking

✨ Features

Intended Use & Task

The model outputs a single probability; values ≥ 0.5 indicate the speaker has completed their utterance.

Model Architecture

• Backbone: wav2vec2 encoder
• Head: shallow linear classifier
• Params: 94.8 M (float32)
• Checkpoint: 360 MB Safetensors (compressed)
  The wav2vec2 + linear configuration out‑performed LSTM and deeper transformer variants during ablation studies.

Training Data

• Sentences were cleaned with Gemini 2.5 Flash to remove ungrammatical, controversial or written‑only text.
• Filler‑word lists per language (e.g., “um”, “えーと”) built with Claude & GPT‑o3 and injected near sentence ends to teach the model about interrupted speech.

All audio/text pairs are released on the pipecat‑ai/datasets hub.

Evaluation & Performance

Accuracy on unseen synthetic test set (50 % complete / 50 % incomplete)

Human English benchmark (human_5_all) : 99 % accuracy.

Inference latency for 8 s audio

oai_citation:7‡Daily

📄 License

This project is licensed under the bsd-2-clause license.

📦 Additional Information