Musicgen Melody

🚀 MusicGen - Melody - 1.5B

Audiocraft offers the code and models for MusicGen, a straightforward and controllable model for music generation. MusicGen simplifies the music - generation process and provides users with more control over the output.

MusicGen is a single - stage auto - regressive Transformer model. It's trained using a 32kHz EnCodec tokenizer with 4 codebooks sampled at 50 Hz. Unlike other methods such as MusicLM, MusicGen doesn't need a self - supervised semantic representation and can generate all 4 codebooks in one pass. By introducing a small delay between the codebooks, it can predict them in parallel, resulting in only 50 auto - regressive steps per second of audio.

MusicGen was introduced in the paper Simple and Controllable Music Generation by Jade Copet, Felix Kreuk, Itai Gat, Tal Remez, David Kant, Gabriel Synnaeve, Yossi Adi, Alexandre Défossez.

Four checkpoints have been released:

• small
• medium
• large
• melody (this checkpoint)

🚀 Quick Start

Try it Online

Run Locally

1. First, install the audiocraft library:

pip install git+https://github.com/facebookresearch/audiocraft.git

2. Ensure that ffmpeg is installed:

apt get install ffmpeg

3. Execute the following Python code:

import torchaudio
from audiocraft.models import MusicGen
from audiocraft.data.audio import audio_write

model = MusicGen.get_pretrained('melody')
model.set_generation_params(duration=8)  # generate 8 seconds.

descriptions = ['happy rock', 'energetic EDM', 'sad jazz']

melody, sr = torchaudio.load('./assets/bach.mp3')
# generates using the melody from the given audio and the provided descriptions.
wav = model.generate_with_chroma(descriptions, melody[None].expand(3, -1, -1), sr)

for idx, one_wav in enumerate(wav):
    # Will save under {idx}.wav, with loudness normalization at -14 db LUFS.
    audio_write(f'{idx}', one_wav.cpu(), model.sample_rate, strategy="loudness")

✨ Features

• Single - stage Generation: MusicGen generates all 4 codebooks in one pass, eliminating the need for a self - supervised semantic representation.
• Parallel Prediction: By introducing a small delay between codebooks, it can predict them in parallel, reducing the number of auto - regressive steps.
• Multiple Checkpoints: Four checkpoints (small, medium, large, and melody) are available for different use - cases.

📚 Documentation

Model Details

Intended Use

• Primary intended use: Research on AI - based music generation, including exploring limitations of generative models and generating music guided by text or melody for learning purposes.
• Primary intended users: Researchers in audio, machine learning, and artificial intelligence, as well as amateurs interested in understanding generative AI models.
• Out - of - scope use cases: Should not be used in downstream applications without risk evaluation. Avoid generating music that creates hostile environments or propagates stereotypes.

Metrics

• Models performance measures:
  • Frechet Audio Distance computed on features from a pre - trained audio classifier (VGGish).
  • Kullback - Leibler Divergence on label distributions from a pre - trained audio classifier (PaSST).
  • CLAP Score between audio and text embeddings from a pre - trained CLAP model.
  • Qualitative studies with human participants on overall quality, text relevance, and melody adherence.
• Decision thresholds: Not applicable.

Evaluation Datasets

The model was evaluated on the MusicCaps benchmark and an in - domain held - out evaluation set with no artist overlap with the training set.

Training Datasets

Trained on licensed data from Meta Music Initiative Sound Collection, Shutterstock music collection, and Pond5 music collection.

Evaluation Results

Limitations and Biases

• Limitations:
  • Can't generate realistic vocals.
  • Performs better with English descriptions.
  • Uneven performance across music styles and cultures.
  • May generate silent endings.
  • Prompt engineering may be needed for satisfying results.
• Biases: The training data may lack diversity, and the model may reflect these biases in its output.
• Risks and harms: Biased or inappropriate samples may be generated.
• Use cases: Users should be aware of biases and limitations and avoid using the model in downstream applications without further investigation.

📄 License

The code is released under the MIT license, and the model weights are released under the CC - BY - NC 4.0 license.