Musicgen Stereo Melody

🚀 MusicGen - Stereo - Melody - 1.5B

MusicGen is a text - to - music model that can generate high - quality music samples based on text descriptions or audio prompts. It simplifies the music generation process and offers multiple pre - trained models for different needs.

🚀 Quick Start

You can try out MusicGen through the following ways:

• You can also run MusicGen locally:

1. First, install the audiocraft library:

pip install git+https://github.com/facebookresearch/audiocraft.git

2. Ensure that ffmpeg is installed:

apt get install ffmpeg

3. Run the following Python code:

import torchaudio
from audiocraft.models import MusicGen
from audiocraft.data.audio import audio_write

model = MusicGen.get_pretrained('melody')
model.set_generation_params(duration=8)  # generate 8 seconds.

descriptions = ['happy rock', 'energetic EDM', 'sad jazz']

melody, sr = torchaudio.load('./assets/bach.mp3')
# generates using the melody from the given audio and the provided descriptions.
wav = model.generate_with_chroma(descriptions, melody[None].expand(3, -1, -1), sr)

for idx, one_wav in enumerate(wav):
    # Will save under {idx}.wav, with loudness normalization at -14 db LUFS.
    audio_write(f'{idx}', one_wav.cpu(), model.sample_rate, strategy="loudness")

✨ Features

• Stereo Capability: We've released a set of stereophonic capable models, fine - tuned from mono models. They work by getting 2 streams of tokens from the EnCodec model and interleaving them using the delay pattern.
• Single - stage Generation: Different from existing methods like MusicLM, MusicGen doesn't require a self - supervised semantic representation and can generate all 4 codebooks in one pass.
• Parallel Prediction: By introducing a small delay between the codebooks, it can predict them in parallel, with only 50 auto - regressive steps per second of audio.

📚 Documentation

Model details

Intended use

• Primary intended use: Research on AI - based music generation, including probing model limitations and generating music guided by text or melody.
• Primary intended users: Researchers in audio, machine learning, and artificial intelligence, as well as machine learning amateurs.
• Out - of - scope use cases: Should not be used in downstream applications without risk evaluation and mitigation. Should not be used to create or disseminate music that creates a hostile or alienating environment.

Metrics

• Models performance measures:
  • Objective measures: Frechet Audio Distance on VGGish features, Kullback - Leibler Divergence on PaSST label distributions, and CLAP Score between audio and text embeddings.
  • Qualitative studies: Evaluated on overall music quality, text relevance, and melody adherence.
• Decision thresholds: Not applicable.

Evaluation datasets

Evaluated on the MusicCaps benchmark and an in - domain held - out evaluation set with no artist overlap with the training set.

Training datasets

Trained on licensed data from Meta Music Initiative Sound Collection, Shutterstock music collection, and Pond5 music collection.

Evaluation results

Limitations and biases

• Data: Trained on 20K hours of data from music professionals. Scaling on larger datasets may improve performance.
• Mitigations: Vocals removed using tags and Hybrid Transformer for Music Source Separation (HT - Demucs).
• Limitations: Can't generate realistic vocals, performs better with English descriptions, varies in performance across music styles and cultures, may generate silent endings, and prompt engineering may be needed.
• Biases: Training data may lack diversity, and generated samples may reflect training data biases.
• Risks and harms: May generate biased, inappropriate, or offensive samples.
• Use cases: Users should be aware of biases, limitations, and risks. Not for downstream applications without further investigation.

📄 License

The code is released under the MIT license, and the model weights are released under the CC - BY - NC 4.0 license.