đ Ultravox
Ultravox is a multimodal Speech LLM that combines speech and text processing capabilities, leveraging powerful pretrained backbones to offer diverse applications.
đ Quick Start
To use the Ultravox model, you can follow the steps below:
import transformers
import numpy as np
import librosa
pipe = transformers.pipeline(model='fixie-ai/ultravox-v0_5-llama-3_2-1b', trust_remote_code=True)
path = "<path-to-input-audio>"
audio, sr = librosa.load(path, sr=16000)
turns = [
{
"role": "system",
"content": "You are a friendly and helpful character. You love to answer questions for people."
},
]
pipe({'audio': audio, 'turns': turns, 'sampling_rate': sr}, max_new_tokens=30)
⨠Features
- Multimodal Input: Ultravox can handle both speech and text as input, enabling more interactive and diverse applications.
- Potential Voice Output: In future revisions, it plans to support the generation of voice output.
- Versatile Applications: It can be used as a voice agent, for speech - to - speech translation, and analysis of spoken audio.
đĻ Installation
Before using the model, you need to install the necessary libraries:
pip install transformers peft librosa
đ Documentation
Model Details
Model Description
Ultravox is a multimodal model that can take both speech and text as input (e.g., a text system prompt and voice user message). The input to the model is given as a text prompt with a special <|audio|>
pseudo - token, and the model processor will replace this magic token with embeddings derived from the input audio. Using the merged embeddings as input, the model will then generate output text as usual.
In a future revision of Ultravox, we plan to expand the token vocabulary to support generation of semantic and acoustic audio tokens, which can then be fed to a vocoder to produce voice output. No preference tuning has been applied to this revision of the model.
- Developed by: Fixie.ai
- License: MIT
Model Sources
- Repository: https://ultravox.ai
- Demo: See repo
Usage
Think of the model as an LLM that can also hear and understand speech. As such, it can be used as a voice agent, and also to do speech - to - speech translation, analysis of spoken audio, etc.
Training Details
Model Backbones
The model uses a pre - trained [Llama3.2 - 1B - Instruct](https://huggingface.co/meta - llama/Meta - Llama - 3.2 - 1B) backbone as well as the encoder part of [whisper - large - v3 - turbo](https://huggingface.co/openai/whisper - large - v3 - turbo).
Training Approach
The multi - modal adapter is trained, the Whisper encoder is fine - tuned, while the Llama model is kept frozen. We use a knowledge - distillation loss where Ultravox is trying to match the logits of the text - based Llama backbone.
Training Data
The training dataset is a mix of ASR datasets, extended with continuations generated by Llama 3.1 8B, and speech translation datasets, which yield a modest improvement in translation evaluations.
Training Procedure
Supervised speech instruction finetuning via knowledge - distillation. For more info, see [training code in Ultravox repo](https://github.com/fixie - ai/ultravox/blob/main/ultravox/training/train.py).
Training Hyperparameters
- Training regime: BF16 mixed precision training
- Hardward used: 8x H100 GPUs
Speeds, Sizes, Times
Check out the audio tab on TheFastest.ai for daily benchmarks and a comparison with other existing models.
Evaluation
|
Ultravox 0.5 1b |
Ultravox 0.5 8B |
Ultravox 0.5 70B |
covost2 en_ar |
1.55 |
12.99 |
20.21 |
covost2 en_ca |
8.06 |
31.54 |
40.01 |
covost2 en_de |
14.21 |
28.70 |
34.53 |
covost2 es_en |
24.97 |
40.19 |
43.29 |
covost2 ru_en |
24.12 |
42.13 |
48.99 |
covost2 zh_en |
4.76 |
17.22 |
21.37 |
big bench audio |
39.14 |
66.54 |
82.70 |
đ License
This project is licensed under the MIT license.