Ultravox V0 3

🚀 Ultravox

Ultravox is a multimodal Speech LLM that combines speech and text input capabilities, offering new possibilities for voice interaction and speech processing.

🚀 Quick Start

Ultravox is a multimodal Speech LLM built around a pretrained Llama3.1-8B-Instruct and Whisper-small backbone. See https://ultravox.ai for the GitHub repo and more information.

✨ Features

• Multimodal Input: Can consume both speech and text as input.
• Future Expansion: Plan to expand the token vocabulary to support generation of semantic and acoustic audio tokens.
• Versatile Usage: Can be used as a voice agent, for speech-to-speech translation, and analysis of spoken audio.

📦 Installation

To use the model, you need to install the necessary libraries:

pip install transformers peft librosa

💻 Usage Examples

Basic Usage

import transformers
import numpy as np
import librosa

pipe = transformers.pipeline(model='fixie-ai/ultravox-v0_3', trust_remote_code=True)

path = "<path-to-input-audio>"  # TODO: pass the audio here
audio, sr = librosa.load(path, sr=16000)


turns = [
  {
    "role": "system",
    "content": "You are a friendly and helpful character. You love to answer questions for people."
  },
]
pipe({'audio': audio, 'turns': turns, 'sampling_rate': sr}, max_new_tokens=30)

📚 Documentation

Model Details

Model Description

Ultravox is a multimodal model that can consume both speech and text as input (e.g., a text system prompt and voice user message). The input to the model is given as a text prompt with a special <|audio|> pseudo-token, and the model processor will replace this magic token with embeddings derived from the input audio. Using the merged embeddings as input, the model will then generate output text as usual.

In a future revision of Ultravox, we plan to expand the token vocabulary to support generation of semantic and acoustic audio tokens, which can then be fed to a vocoder to produce voice output. No preference tuning has been applied to this revision of the model.

Training Details

Training Data

Training dataset is a mix of ASR datasets, extended by adding a "continuation" generated by Llama 3.1 8B.

Training Procedure

Supervised speech to audio finetuning. For more info, see training code in Ultravox repo.

Training Hyperparameters

• Training regime: BF16 mixed precision training
• Hardward used: 8x H100 GPUs

Speeds, Sizes, Times

The current version of Ultravox, when invoked with audio content, has a time-to-first-token (TTFT) of approximately 200ms, and a tokens-per-second rate of ~50-100 when using an A100-40GB GPU, all using a Llama 3.1 8B backbone.

Check out the audio tab on TheFastest.ai for daily benchmarks and a comparison with other existing models.

Evaluation

📄 License

This model is released under the MIT license.