Musicgen Medium

🚀 MusicGen - Medium - 1.5B

MusicGen is a text-to-music model that can generate high-quality music samples based on text descriptions or audio prompts. It simplifies the music generation process and offers more efficient and controllable music creation.

🚀 Quick Start

Try Online

You can try out MusicGen yourself through the following links:

• Audiocraft Colab:
• Hugging Face Colab:
• Hugging Face Demo:

Run Locally

You can run MusicGen locally using either the 🤗 Transformers library or the Audiocraft library.

🤗 Transformers Usage

1. Installation First, install the 🤗 Transformers library and scipy:

pip install --upgrade pip
pip install --upgrade transformers scipy

2. Inference via the Text-to-Audio (TTA) pipeline

from transformers import pipeline
import scipy

synthesiser = pipeline("text-to-audio", "facebook/musicgen-medium")

music = synthesiser("lo-fi music with a soothing melody", forward_params={"do_sample": True})

scipy.io.wavfile.write("musicgen_out.wav", rate=music["sampling_rate"], data=music["audio"])

3. Inference via the Transformers modelling code

from transformers import AutoProcessor, MusicgenForConditionalGeneration

processor = AutoProcessor.from_pretrained("facebook/musicgen-medium")
model = MusicgenForConditionalGeneration.from_pretrained("facebook/musicgen-medium")

inputs = processor(
    text=["80s pop track with bassy drums and synth", "90s rock song with loud guitars and heavy drums"],
    padding=True,
    return_tensors="pt",
)

audio_values = model.generate(**inputs, max_new_tokens=256)

4. Listen to the audio samples In an ipynb notebook:

from IPython.display import Audio

sampling_rate = model.config.audio_encoder.sampling_rate
Audio(audio_values[0].numpy(), rate=sampling_rate)

Or save them as a .wav file using scipy:

import scipy

sampling_rate = model.config.audio_encoder.sampling_rate
scipy.io.wavfile.write("musicgen_out.wav", rate=sampling_rate, data=audio_values[0, 0].numpy())

Audiocraft Usage

1. Installation First, install the audiocraft library:

pip install git+https://github.com/facebookresearch/audiocraft.git

2. Install ffmpeg

apt-get install ffmpeg

3. Run the Python code

from audiocraft.models import MusicGen
from audiocraft.data.audio import audio_write

model = MusicGen.get_pretrained("medium")
model.set_generation_params(duration=8)  # generate 8 seconds.

descriptions = ["happy rock", "energetic EDM"]

wav = model.generate(descriptions)  # generates 2 samples.

for idx, one_wav in enumerate(wav):
    # Will save under {idx}.wav, with loudness normalization at -14 db LUFS.
    audio_write(f'{idx}', one_wav.cpu(), model.sample_rate, strategy="loudness")

✨ Features

• Single-stage Generation: Unlike some existing methods, MusicGen doesn't require a self - supervised semantic representation and generates all 4 codebooks in one pass.
• Parallel Prediction: By introducing a small delay between the codebooks, it can predict them in parallel, reducing the number of auto - regressive steps per second of audio.

📚 Documentation

Model details

Intended use

• Primary intended use: Research on AI - based music generation, including exploring model limitations and generating music guided by text or melody.
• Primary intended users: Researchers in audio, machine learning, and artificial intelligence, as well as amateurs interested in understanding generative models.
• Out - of - scope use cases: Should not be used in downstream applications without further risk evaluation and mitigation. Avoid using it to create or disseminate offensive or biased music.

Metrics

• Models performance measures:
  • Objective measures: Frechet Audio Distance, Kullback - Leibler Divergence, CLAP Score.
  • Qualitative studies: Evaluated on overall music quality, text relevance, and adherence to melody.
• Decision thresholds: Not applicable.

Evaluation datasets

Evaluated on the MusicCaps benchmark and an in - domain held - out evaluation set with no artist overlap with the training set.

Training datasets

Trained on licensed data from Meta Music Initiative Sound Collection, Shutterstock music collection, and Pond5 music collection.

Evaluation results

Limitations and biases

• Data: The training data may lack diversity, and not all music cultures are equally represented.
• Limitations: Can't generate realistic vocals, performs better with English descriptions, may not work well for all music styles, sometimes generates silent endings, and prompt engineering may be needed.
• Biases: Generated samples may reflect biases in the training data.
• Risks and harms: Biased or inappropriate samples may be generated.
• Use cases: Users should be aware of the risks and not use the model in downstream applications without further investigation.

⚠️ Important Note

Biases and limitations of the model may lead to generation of samples that may be considered as biased, inappropriate or offensive. Users must be aware of these issues and should not use MusicGen for downstream applications without further investigation and mitigation of risks.

💡 Usage Tip

Prompt engineering may be required to obtain satisfying results when using MusicGen. Try different text descriptions to get the best music generation.