AudioX Open-Source Audio Model - Freely Convert Any Content to High-Quality Music and General Audio

Audiox

Developed by HKUSTAudio

AudioX is a unified diffusion transformer model capable of generating audio and music from arbitrary content. It produces high-quality general audio and musical compositions, offers flexible natural language control, and seamlessly handles multimodal inputs.

Text-to-Audio #Multimodal audio generation #Diffusion transformer #Video soundtrack generation

Downloads 2,189

Release Time : 4/2/2025

Model Overview

AudioX is a multimodal audio generation model that can convert various inputs such as text, video, images, music, and audio into high-quality audio or musical compositions.

Model Features

Multimodal input support

Capable of processing various input modalities including text, video, images, music, and audio

High-quality audio generation

Generates professional-grade general audio and musical compositions

Natural language control

Flexibly controls audio generation content and style through text prompts

Unified architecture

Uses a diffusion transformer architecture to uniformly handle different audio generation tasks

Model Capabilities

Text-to-audio generation

Video soundtrack generation

Image-to-audio conversion

Audio style transfer

Music composition

Use Cases

Multimedia creation

Video soundtrack generation

Automatically generates background music that matches the video

Produces professional-grade soundtracks that harmonize with the video content

Sound effect design

Generates specific scene sound effects based on text descriptions

Creates realistic environmental and special sound effects

Music creation

Music generation

Creates complete musical compositions based on text prompts

Generates music with specific styles and emotions

Music adaptation

Transforms existing music into different styles

Changes the musical style while preserving the original structure

🚀 AudioX

AudioX is a unified Diffusion Transformer model for Anything-to-Audio and Music Generation, capable of generating high-quality general audio and music, offering flexible natural language control, and seamlessly processing various modalities including text, video, image, music, and audio.

🚀 Quick Start

📦 Installation

GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/HKUSTAudio/AudioX
cd AudioX

conda create -n AudioX python=3.8.20
conda activate AudioX
pip install git+https://github.com/ZeyueT/AudioX.git
conda install -c conda-forge ffmpeg libsndfile

💻 Usage Examples

Basic Usage

import torch
import torchaudio
from einops import rearrange
from stable_audio_tools import get_pretrained_model
from stable_audio_tools.inference.generation import generate_diffusion_cond
from stable_audio_tools.data.utils import read_video, merge_video_audio
from stable_audio_tools.data.utils import load_and_process_audio
import os

device = "cuda" if torch.cuda.is_available() else "cpu"

# Download model
model, model_config = get_pretrained_model("HKUSTAudio/AudioX")
sample_rate = model_config["sample_rate"]
sample_size = model_config["sample_size"]
target_fps = model_config["video_fps"]
seconds_start = 0
seconds_total = 10

model = model.to(device)

# for video-to-music generation
video_path = "video.mp4"
text_prompt = "Generate music for the video" 
audio_path = None

video_tensor = read_video(video_path, seek_time=0, duration=seconds_total, target_fps=target_fps)
audio_tensor = load_and_process_audio(audio_path, sample_rate, seconds_start, seconds_total)

conditioning = [{
    "video_prompt": [video_tensor.unsqueeze(0)],        
    "text_prompt": text_prompt,
    "audio_prompt": audio_tensor.unsqueeze(0),
    "seconds_start": seconds_start,
    "seconds_total": seconds_total
}]
    
# Generate stereo audio
output = generate_diffusion_cond(
    model,
    steps=250,
    cfg_scale=7,
    conditioning=conditioning,
    sample_size=sample_size,
    sigma_min=0.3,
    sigma_max=500,
    sampler_type="dpmpp-3m-sde",
    device=device
)

# Rearrange audio batch to a single sequence
output = rearrange(output, "b d n -> d (b n)")

# Peak normalize, clip, convert to int16, and save to file
output = output.to(torch.float32).div(torch.max(torch.abs(output))).clamp(-1, 1).mul(32767).to(torch.int16).cpu()
torchaudio.save("output.wav", output, sample_rate)

if video_path is not None and os.path.exists(video_path):
    merge_video_audio(video_path, "output.wav", "output.mp4", 0, seconds_total)

📚 Documentation

If you find our work useful, please consider citing:

@article{tian2025audiox,
  title={AudioX: Diffusion Transformer for Anything-to-Audio Generation},
  author={Tian, Zeyue and Jin, Yizhu and Liu, Zhaoyang and Yuan, Ruibin and Tan, Xu and Chen, Qifeng and Xue, Wei and Guo, Yike},
  journal={arXiv preprint arXiv:2503.10522},
  year={2025}
}