Musicgen Stereo Medium

🚀 MusicGen - Stereo - Medium - 1.5B

MusicGen is a text - to - music model that can generate high - quality music samples based on text descriptions or audio prompts. This release focuses on stereophonic capable models, which offer a more immersive audio experience.

🚀 Quick Start

MusicGen is a powerful text - to - music model. The stereophonic models are fine - tuned from mono models, sharing capabilities and limitations with the base models. They work by getting 2 streams of tokens from the EnCodec model and interleaving them using a delay pattern.

✨ Features

• Stereophonic Sound: Reproduces sound with depth and direction, creating a more immersive audio experience.
• Single - stage Auto - regressive: Generates all 4 codebooks in one pass without the need for a self - supervised semantic representation.
• Multiple Pre - trained Models: Offers 10 pre - trained models, including small, medium, large, and melody - guided variants.

📦 Installation

You can run MusicGen Stereo models locally with the 🤗 Transformers library from main onward.

1. First install the 🤗 Transformers library and scipy:

pip install --upgrade pip
pip install --upgrade git+https://github.com/huggingface/transformers.git scipy

💻 Usage Examples

Basic Usage

Run inference via the Text - to - Audio (TTA) pipeline:

import torch
import soundfile as sf
from transformers import pipeline

synthesiser = pipeline("text - to - audio", "facebook/musicgen - stereo - medium", device="cuda:0", torch_dtype=torch.float16)

music = synthesiser("lo - fi music with a soothing melody", forward_params={"max_new_tokens": 256})

sf.write("musicgen_out.wav", music["audio"][0].T, music["sampling_rate"])

Advanced Usage

Run inference via the Transformers modelling code for more fine - grained control:

from transformers import AutoProcessor, MusicgenForConditionalGeneration

processor = AutoProcessor.from_pretrained("facebook/musicgen - stereo - medium")
model = MusicgenForConditionalGeneration.from_pretrained("facebook/musicgen - stereo - medium").to("cuda")

inputs = processor(
    text=["80s pop track with bassy drums and synth", "90s rock song with loud guitars and heavy drums"],
    padding=True,
    return_tensors="pt",
).to("cuda")

audio_values = model.generate(**inputs, max_new_tokens=256)

📚 Documentation

Model Details

@misc{copet2023simple,
      title={Simple and Controllable Music Generation}, 
      author={Jade Copet and Felix Kreuk and Itai Gat and Tal Remez and David Kant and Gabriel Synnaeve and Yossi Adi and Alexandre Défossez},
      year={2023},
      eprint={2306.05284},
      archivePrefix={arXiv},
      primaryClass={cs.SD}
}

| License | Code is released under MIT, model weights are released under CC - BY - NC 4.0 | | Where to send questions or comments about the model | Via the Github repository of the project or by opening an issue |

Intended Use

Primary intended use:

• Research on AI - based music generation, such as probing and understanding the limitations of generative models.
• Generation of music guided by text or melody for machine learning amateurs to understand generative AI models.

Primary intended users: Researchers in audio, machine learning, and artificial intelligence, as well as amateurs interested in these models.

Out - of - scope use cases: The model should not be used on downstream applications without further risk evaluation and mitigation. It should not be used to create or disseminate music that creates hostile or offensive environments.

Metrics

Models performance measures:

• Frechet Audio Distance computed on features from a pre - trained audio classifier (VGGish).
• Kullback - Leibler Divergence on label distributions from a pre - trained audio classifier (PaSST).
• CLAP Score between audio and text embeddings from a pre - trained CLAP model.

Additionally, qualitative studies with human participants evaluate the model on overall quality, text relevance, and adherence to melody.

Evaluation Datasets

The model was evaluated on the MusicCaps benchmark and an in - domain held - out evaluation set with no artist overlap with the training set.

Training Datasets

The model was trained on licensed data from the Meta Music Initiative Sound Collection, Shutterstock music collection, and Pond5 music collection.

Evaluation Results

Limitations and Biases

Data: The model is trained on 20K hours of data from professional music sources. Scaling on larger datasets may improve performance. Mitigations: Vocals are removed using tags and the Hybrid Transformer for Music Source Separation (HT - Demucs). Limitations:

• Unable to generate realistic vocals.
• Performs better with English descriptions.
• Uneven performance across different music styles and cultures.
• Sometimes generates song endings that turn to silence.
• Prompt engineering may be needed for satisfying results. Biases: The data source may lack diversity.

📄 License

Code is released under MIT, model weights are released under CC - BY - NC 4.0.