๐ Ultravox
Ultravox is a multimodal Speech LLM that combines a pretrained LLM with a speech encoder, enabling it to handle both speech and text inputs.
๐ Quick Start
Think of the model as an LLM that can also hear and understand speech. As such, it can be used as a voice agent, and also to do speech - to - speech translation, analysis of spoken audio, etc.
To use the model, try the following:
import transformers
import numpy as np
import librosa
pipe = transformers.pipeline(model='fixie-ai/ultravox-v0_6-llama-3_1-8b', trust_remote_code=True)
path = "<path-to-input-audio>"
audio, sr = librosa.load(path, sr=16000)
turns = [
{
"role": "system",
"content": "You are a friendly and helpful character. You love to answer questions for people."
},
]
pipe({'audio': audio, 'turns': turns, 'sampling_rate': sr}, max_new_tokens=30)
โจ Features
- Can consume both speech and text as input.
- In v0.6 series, it has improved Hindi speech understanding performance and better noise robustness.
- Plans to expand the token vocabulary for voice output in the future.
๐ฆ Installation
No specific installation steps provided in the original README.
๐ Documentation
๐ Model Details
๐ Model Description
Ultravox is a multimodal model that can consume both speech and text as input (e.g., a text system prompt and voice user message).
The input to the model is given as a text prompt with a special <|audio|>
pseudo - token, and the model processor will replace this magic token with embeddings derived from the input audio. Using the merged embeddings as input, the model will then generate output text as usual.
In v0.6 series, ultravox models are trained on expanded Hindi speech data, resulting in significantly improved speech understanding performance on Hindi and modest degradation on other languages. Additionally, the v0.6 models are also trained on noise datasets for improved noise robustness and the ability to output a special string ((noise))
if the input audio is too noisy or doesn't contain clear speech.
In a future revision of Ultravox, we plan to expand the token vocabulary to support generation of semantic and acoustic audio tokens, which can then be fed to a vocoder to produce voice output.
No preference tuning has been applied to this revision of the model.
- Developed by: Fixie.ai
- License: MIT
๐ Model Sources
- Repository: https://ultravox.ai
- Demo: See repo
๐ ๏ธ Training Details
๐ Training Data
The training dataset is a mix of ASR datasets, extended with continuations generated by Llama 3.1 8B, speech translation datasets, and noise datasets.
๐ Training Procedure
Supervised speech instruction finetuning via knowledge - distillation. For more info, see training code in Ultravox repo.
โ๏ธ Training Hyperparameters
- Training regime: BF16 mixed precision training
- Hardware used: 8x H100 GPUs
โฑ๏ธ Speeds, Sizes, Times
The current version of Ultravox, when invoked with audio content, has a time - to - first - token (TTFT) of approximately 150ms, and a tokens - per - second rate of ~50 - 100 when using an A100 - 40GB GPU, all using a text - based LLM (Llama, Gemma, or Qwen) backbone.
Check out the audio tab on TheFastest.ai for daily benchmarks and a comparison with other existing models.
๐งช Evaluation
Evaluations are conducted on covost2 (speech translation measured in BLEU), fleurs and ultravox_calls (speech recognition measured in WER), big bench audio (audio reasoning measured in accuracy), as well as musan and ultravox_unintelligible (noise/unintelligible speech detection measured in recall).
|
v0_5 - llama - 3_1 - 8b |
v0_6 - llama - 3_1 - 8b |
v0_5 - llama - 3_3 - 70b |
v0_6 - llama - 3_3 - 70b |
v0_6 - gemma - 3 - 27b |
v0_6 - qwen - 3 - 32b |
covost2 en_ar |
12.90 |
12.94 |
20.21 |
18.92 |
22.68 |
16.91 |
covost2 en_ca |
31.51 |
31.47 |
40.01 |
38.73 |
39.67 |
33.63 |
covost2 en_de |
28.60 |
28.66 |
34.53 |
33.69 |
34.76 |
31.09 |
covost2 es_en |
40.41 |
40.36 |
43.29 |
41.39 |
41.11 |
41.20 |
covost2 ru_en |
42.22 |
42.41 |
48.99 |
43.73 |
49.29 |
47.08 |
covost2 zh_en |
16.97 |
17.24 |
21.37 |
17.81 |
20.88 |
22.24 |
librispeech |
2.04 |
2.09 |
2.09 |
2.55 |
2.73 |
2.88 |
fleurs cmn_hans_cn |
12.11 |
12.25 |
11.20 |
13.49 |
12.56 |
12.10 |
fleurs de_de |
6.66 |
7.56 |
5.26 |
7.14 |
4.86 |
6.83 |
fleurs es_419 |
5.74 |
5.83 |
4.53 |
6.06 |
4.68 |
5.14 |
fleurs hi_in |
29.74 |
10.34 |
18.90 |
11.43 |
8.40 |
11.78 |
ultravox_calls (asr) |
22.31 |
20.01 |
19.56 |
16.51 |
19.56 |
28.67 |
big bench audio |
68.06 |
69.70 |
90.15 |
85.48 |
83.84 |
84.22 |
musan_noise |
0.00 |
97.45 |
0.00 |
98.51 |
99.58 |
99.78 |
ultravox_unintelligible |
0.00 |
45.78 |
0.00 |
50.00 |
66.84 |
64.21 |
๐ License
The model is licensed under the MIT license.