Riffusion Model V1

🚀 Riffusion

Riffusion is an app for real-time music generation with stable diffusion. It enables users to generate music in real - time. You can learn more about it at https://www.riffusion.com/about and have a try at https://www.riffusion.com/.

• Code: https://github.com/riffusion/riffusion
• Web app: https://github.com/hmartiro/riffusion-app
• Model checkpoint: https://huggingface.co/riffusion/riffusion-model-v1
• Discord: https://discord.gg/yu6SRwvX4v

This repository contains the model files, including a diffusers formated library, a compiled checkpoint file, a traced unet for improved inference speed, and a seed image library for use with riffusion - app.

✨ Features

• Real - time music generation using stable diffusion.
• Capable of generating spectrogram images from text input, which can be converted into audio clips.

📦 Installation

No installation steps are provided in the original README, so this section is skipped.

💻 Usage Examples

No code examples are provided in the original README, so this section is skipped.

📚 Documentation

Riffusion v1 Model

Riffusion is a latent text - to - image diffusion model. It can generate spectrogram images from any text input, and these spectrograms can be converted into audio clips.

The model was created by Seth Forsgren and Hayk Martiros as a hobby project. You can either use the Riffusion model directly or try the Riffusion web app.

The Riffusion model was created by fine - tuning the Stable - Diffusion - v1 - 5 checkpoint. You can read about Stable Diffusion in 🤗's Stable Diffusion blog.

Model Details

Direct Use

The model is for research purposes only. Possible research areas and tasks include:

• Generation of artworks, audio, and use in creative processes.
• Applications in educational or creative tools.
• Research on generative models.

Datasets

The original Stable Diffusion v1.5 was trained on the LAION - 5B dataset using the CLIP text encoder. It provides a great starting point with an in - depth understanding of language, including musical concepts. The team at LAION also compiled a great audio dataset from many general, speech, and music sources, which is recommended at [LAION - AI/audio - dataset](https://github.com/LAION - AI/audio - dataset/blob/main/data_collection/README.md).

Fine Tuning

Check out the diffusers training examples from Hugging Face. Fine tuning requires a dataset of spectrogram images of short audio clips, with associated text describing them. Note that the CLIP encoder can understand and connect many words even if they never appear in the dataset. It is also possible to use a dreambooth method to get custom styles.

License Information

⚠️ Important Note

This model is open access and available to all, with a CreativeML OpenRAIL - M license further specifying rights and usage. The CreativeML OpenRAIL License specifies:

1. You can't use the model to deliberately produce nor share illegal or harmful outputs or content.
2. Riffusion claims no rights on the outputs you generate, you are free to use them and are accountable for their use which must not go against the provisions set in the license.
3. You may re - distribute the weights and use the model commercially and/or as a service. If you do, please be aware you have to include the same use restrictions as the ones in the license and share a copy of the CreativeML OpenRAIL - M to all your users (please read the license entirely and carefully). Please read the full license carefully here: https://huggingface.co/spaces/CompVis/stable - diffusion - license

🔧 Technical Details

The model is a latent text - to - image diffusion model. It is based on fine - tuning the Stable - Diffusion - v1 - 5 checkpoint. It uses a fixed, pretrained text encoder (CLIP ViT - L/14) as suggested in the Imagen paper. The original Stable Diffusion v1.5 was trained on the LAION - 5B dataset with the CLIP text encoder. Fine - tuning requires a dataset of spectrogram images of short audio clips and associated text descriptions.

📄 License

This model is released under the [CreativeML OpenRAIL M license](https://huggingface.co/spaces/CompVis/stable - diffusion - license).

📖 Citation

If you build on this work, please cite it as follows:

@article{Forsgren_Martiros_2022,
  author = {Forsgren, Seth* and Martiros, Hayk*},
  title = {{Riffusion - Stable diffusion for real-time music generation}},
  url = {https://riffusion.com/about},
  year = {2022}
}