Musicgen Stereo Melody Large

🚀 MusicGen - Stereo - Melody - Large 3.3B

MusicGen is a text-to-music model that can generate high - quality music samples based on text descriptions or audio prompts. It simplifies the music generation process and offers multiple pre - trained models for different use cases. This release includes stereophonic capable models fine - tuned from mono models, sharing similar capabilities and limitations with the base models.

🚀 Quick Start

You can try out MusicGen in multiple ways:

• Online demos:

  • 
  -

• Local run:

  1. First, install the audiocraft library:

  pip install git+https://github.com/facebookresearch/audiocraft.git
  

  2. Make sure to have ffmpeg installed:

  apt get install ffmpeg
  

  3. Run the following Python code:

  import torchaudio
  from audiocraft.models import MusicGen
  from audiocraft.data.audio import audio_write
  
  model = MusicGen.get_pretrained('melody')
  model.set_generation_params(duration=8)  # generate 8 seconds.
  
  descriptions = ['happy rock', 'energetic EDM', 'sad jazz']
  
  melody, sr = torchaudio.load('./assets/bach.mp3')
  # generates using the melody from the given audio and the provided descriptions.
  wav = model.generate_with_chroma(descriptions, melody[None].expand(3, -1, -1), sr)
  
  for idx, one_wav in enumerate(wav):
      # Will save under {idx}.wav, with loudness normalization at -14 db LUFS.
      audio_write(f'{idx}', one_wav.cpu(), model.sample_rate, strategy="loudness")
  

✨ Features

• Stereophonic models: A set of stereophonic capable models are released, fine - tuned from mono models for 200k updates.
• Single - stage generation: Unlike some existing methods, MusicGen doesn't require a self - supervised semantic representation and generates all 4 codebooks in one pass.
• Parallel prediction: By introducing a small delay between the codebooks, it can predict them in parallel, with only 50 auto - regressive steps per second of audio.
• Multiple pre - trained models: There are 10 pre - trained models available, including different sizes (300M, 1.5B, 3.3B parameters) and variants (text - to - music and melody - guided music generation).

💻 Usage Examples

Basic Usage

import torchaudio
from audiocraft.models import MusicGen
from audiocraft.data.audio import audio_write

model = MusicGen.get_pretrained('melody')
model.set_generation_params(duration=8)  # generate 8 seconds.

descriptions = ['happy rock', 'energetic EDM', 'sad jazz']

melody, sr = torchaudio.load('./assets/bach.mp3')
# generates using the melody from the given audio and the provided descriptions.
wav = model.generate_with_chroma(descriptions, melody[None].expand(3, -1, -1), sr)

for idx, one_wav in enumerate(wav):
    # Will save under {idx}.wav, with loudness normalization at -14 db LUFS.
    audio_write(f'{idx}', one_wav.cpu(), model.sample_rate, strategy="loudness")

📚 Documentation

Model details

Intended use

• Primary intended use: Research on AI - based music generation, including probing model limitations and generating music guided by text or melody.
• Primary intended users: Researchers in audio, machine learning and artificial intelligence, as well as amateurs interested in understanding these models.
• Out - of - scope use cases: The model should not be used in downstream applications without further risk evaluation and mitigation. It should not be used to create or disseminate offensive music.

Metrics

• Models performance measures:
  • Frechet Audio Distance computed on features from a pre - trained audio classifier (VGGish).
  • Kullback - Leibler Divergence on label distributions from a pre - trained audio classifier (PaSST).
  • CLAP Score between audio embedding and text embedding from a pre - trained CLAP model.
  • Qualitative studies with human participants on music quality, text relevance, and melody adherence.
• Decision thresholds: Not applicable.

Evaluation datasets

The model was evaluated on the MusicCaps benchmark and an in - domain held - out evaluation set with no artist overlap with the training set.

Training datasets

The model was trained on licensed data from Meta Music Initiative Sound Collection, Shutterstock music collection and Pond5 music collection.

Evaluation results

Limitations and biases

• Data: The model was trained on 20K hours of data from music professionals. Scaling on larger datasets may improve performance.
• Mitigations: Vocals were removed using tags and Hybrid Transformer for Music Source Separation (HT - Demucs).
• Limitations:
  • Unable to generate realistic vocals.
  • Performs better with English descriptions.
  • Uneven performance across different music styles and cultures.
  • Sometimes generates silent endings.
  • Difficult to determine optimal text descriptions.
• Biases: The data source may lack diversity, and generated samples may reflect training data biases.
• Risks and harms: Biased or inappropriate samples may be generated.
• Use cases: Users should be aware of biases, limitations and risks. It should not be used for downstream applications without further investigation.

📄 License

Code is released under MIT, model weights are released under CC - BY - NC 4.0.