Musicgen Stereo Small

🚀 MusicGen - Stereo - Small - 300M

MusicGen is a text - to - music model that can generate high - quality music samples based on text descriptions or audio prompts. This stereo - capable model is fine - tuned from mono models, offering a new dimension to music generation.

🚀 Quick Start

MusicGen is a powerful text - to - music model. You can quickly start using it through the 🤗 Transformers library or the Audiocraft library.

✨ Features

• Stereo Capability: Fine - tuned from mono models, it can generate stereophonic music, adding depth and direction to the sound.
• Single - stage Generation: Unlike some existing methods, it doesn't require a self - supervised semantic representation and can generate all 4 codebooks in one pass.
• Multiple Pre - trained Models: Offers 10 pre - trained models with different sizes and capabilities, suitable for various scenarios.

📦 Installation

You can run MusicGen Stereo models locally with the 🤗 Transformers library from main onward.

1. First install the 🤗 Transformers library and scipy:

pip install --upgrade pip
pip install --upgrade git+https://github.com/huggingface/transformers.git scipy

2. If using the Audiocraft library, first install the audiocraft library:

pip install git+https://github.com/facebookresearch/audiocraft.git

3. Make sure to have ffmpeg installed:

apt get install ffmpeg

💻 Usage Examples

Basic Usage

Run inference via the Text - to - Audio (TTA) pipeline:

import torch
import soundfile as sf
from transformers import pipeline

synthesiser = pipeline("text - to - audio", "facebook/musicgen - stereo - small", device="cuda:0", torch_dtype=torch.float16)

music = synthesiser("lo - fi music with a soothing melody", forward_params={"max_new_tokens": 256})

sf.write("musicgen_out.wav", music["audio"][0].T, music["sampling_rate"])

Advanced Usage

Run inference via the Transformers modelling code for more fine - grained control:

from transformers import AutoProcessor, MusicgenForConditionalGeneration

processor = AutoProcessor.from_pretrained("facebook/musicgen - stereo - small")
model = MusicgenForConditionalGeneration.from_pretrained("facebook/musicgen - stereo - small").to("cuda")

inputs = processor(
    text=["80s pop track with bassy drums and synth", "90s rock song with loud guitars and heavy drums"],
    padding=True,
    return_tensors="pt",
).to("cuda")

audio_values = model.generate(**inputs, max_new_tokens=256)

📚 Documentation

Model Details

Intended Use

• Primary intended use: Research on AI - based music generation, including probing limitations of generative models and generating music guided by text or melody.
• Primary intended users: Researchers in audio, machine learning, and artificial intelligence, as well as amateurs interested in understanding these models.
• Out - of - scope use cases: Should not be used in downstream applications without risk evaluation and mitigation. Should not be used to create or disseminate music that creates hostile or alienating environments.

Metrics

• Models performance measures:
  • Frechet Audio Distance computed on features from a pre - trained audio classifier (VGGish).
  • Kullback - Leibler Divergence on label distributions from a pre - trained audio classifier (PaSST).
  • CLAP Score between audio embedding and text embedding from a pre - trained CLAP model.
  • Qualitative studies with human participants on overall quality, text relevance, and melody adherence.
• Decision thresholds: Not applicable.

Evaluation Datasets

Evaluated on the MusicCaps benchmark and an in - domain held - out evaluation set with no artist overlap with the training set.

Training Datasets

Trained on licensed data from Meta Music Initiative Sound Collection, Shutterstock music collection, and Pond5 music collection.

Evaluation Results

Limitations and Biases

• Data: The data sources may lack diversity. The model was trained on 20K hours of data, and scaling on larger datasets may improve performance.
• Mitigations: Vocals were removed from the data source using tags and a state - of - the - art music source separation method (Hybrid Transformer for Music Source Separation (HT - Demucs)).
• Limitations: Can't generate realistic vocals, performs better with English descriptions, may not work equally well for all music styles and cultures, sometimes generates silent endings, and prompt engineering may be required for satisfying results.

🔧 Technical Details

MusicGen is a single - stage auto - regressive Transformer model trained over a 32kHz EnCodec tokenizer with 4 codebooks sampled at 50 Hz. The stereo models work by getting 2 streams of tokens from the EnCodec model and interleaving them using the delay pattern.

📄 License

Code is released under MIT, model weights are released under CC - BY - NC 4.0.