Musicgen Melody Large

🚀 MusicGen - Melody - Large 3.3B

MusicGen is a text-to-music model that can generate high-quality music samples based on text descriptions or audio prompts. It simplifies the music generation process and offers a more efficient way to create music with specific characteristics.

🚀 Quick Start

You can try out MusicGen in multiple ways:

• To run MusicGen locally, follow these steps:

📦 Installation

1. First, install the audiocraft library:

pip install git+https://github.com/facebookresearch/audiocraft.git

2. Ensure that ffmpeg is installed:

apt get install ffmpeg

💻 Usage Examples

Basic Usage

import torchaudio
from audiocraft.models import MusicGen
from audiocraft.data.audio import audio_write

model = MusicGen.get_pretrained('melody')
model.set_generation_params(duration=8)  # generate 8 seconds.

descriptions = ['happy rock', 'energetic EDM', 'sad jazz']

melody, sr = torchaudio.load('./assets/bach.mp3')
# generates using the melody from the given audio and the provided descriptions.
wav = model.generate_with_chroma(descriptions, melody[None].expand(3, -1, -1), sr)

for idx, one_wav in enumerate(wav):
    # Will save under {idx}.wav, with loudness normalization at -14 db LUFS.
    audio_write(f'{idx}', one_wav.cpu(), model.sample_rate, strategy="loudness")

✨ Features

• Single-stage Generation: Unlike some existing methods such as MusicLM, MusicGen doesn't require a self-supervised semantic representation and generates all 4 codebooks in one pass.
• Parallel Prediction: By introducing a small delay between the codebooks, it can predict them in parallel, with only 50 auto-regressive steps per second of audio.
• Multiple Pre-trained Models: We offer 10 pre-trained models, including different sizes and variants for text-to-music and text+melody-to-music generation.

The pre trained models are:

• facebook/musicgen-small: 300M model, text to music only - 🤗 Hub
• facebook/musicgen-medium: 1.5B model, text to music only - 🤗 Hub
• facebook/musicgen-melody: 1.5B model, text to music and text+melody to music - 🤗 Hub
• facebook/musicgen-large: 3.3B model, text to music only - 🤗 Hub
• facebook/musicgen-melody-large: 3.3B model, text to music and text+melody to music - 🤗 Hub
• facebook/musicgen-stereo-*: All the previous models fine-tuned for stereo generation - small, medium, large, melody, melody large

📚 Documentation

Model details

Intended use

• Primary intended use: The primary use of MusicGen is research on AI-based music generation, including research efforts to understand generative model limitations and generating music guided by text or melody for machine learning amateurs.
• Primary intended users: Researchers in audio, machine learning and artificial intelligence, as well as amateurs seeking to better understand those models.
• Out-of-scope use cases: The model should not be used on downstream applications without further risk evaluation and mitigation. It should not be used to create or disseminate music that creates hostile or alienating environments for people.

Metrics

• Models performance measures:
  • Objective measures: Frechet Audio Distance computed on features extracted from a pre-trained audio classifier (VGGish), Kullback-Leibler Divergence on label distributions extracted from a pre-trained audio classifier (PaSST), CLAP Score between audio embedding and text embedding extracted from a pre-trained CLAP model.
  • Qualitative studies: Overall quality of the music samples, text relevance to the provided text input, adherence to the melody for melody-guided music generation.
• Decision thresholds: Not applicable.

Evaluation datasets

The model was evaluated on the MusicCaps benchmark and an in-domain held-out evaluation set with no artist overlap with the training set.

Training datasets

The model was trained on licensed data from the Meta Music Initiative Sound Collection, Shutterstock music collection and the Pond5 music collection.

Evaluation results

Limitations and biases

• Data: The data sources are created by music professionals and covered by legal agreements. Scaling the model on larger datasets may improve performance.
• Mitigations: Vocals have been removed using tags and the Hybrid Transformer for Music Source Separation (HT-Demucs).
• Limitations:
  • Unable to generate realistic vocals.
  • Performs better with English descriptions.
  • Uneven performance across different music styles and cultures.
  • Sometimes generates end-of-song silence.
  • Difficult to determine optimal text descriptions, may require prompt engineering.
• Biases: The data source may lack diversity, and the model may not perform equally well on all music genres. Generated samples may reflect training data biases.
• Risks and harms: Biases and limitations may lead to the generation of inappropriate or offensive samples.

⚠️ Important Note

Users must be aware of the biases, limitations and risks of the model. MusicGen is developed for artificial intelligence research on controllable music generation and should not be used for downstream applications without further investigation and risk mitigation.

MusicGen was published in Simple and Controllable Music Generation by Jade Copet, Felix Kreuk, Itai Gat, Tal Remez, David Kant, Gabriel Synnaeve, Yossi Adi, Alexandre Défossez.