Mms Tts Guj

🚀 Massively Multilingual Speech (MMS): Gujarati Text-to-Speech

This repository offers a text-to-speech (TTS) model checkpoint for the Gujarati (guj) language. It's part of Facebook's Massively Multilingual Speech project, aiming to provide speech technology across a wide range of languages.

🚀 Quick Start

MMS-TTS has been available in the 🤗 Transformers library since version 4.33. To use this checkpoint, first install the latest version of the library:

pip install --upgrade transformers accelerate

Then, run inference with the following code-snippet:

from transformers import VitsModel, AutoTokenizer
import torch

model = VitsModel.from_pretrained("facebook/mms-tts-guj")
tokenizer = AutoTokenizer.from_pretrained("facebook/mms-tts-guj")

text = "some example text in the Gujarati language"
inputs = tokenizer(text, return_tensors="pt")

with torch.no_grad():
    output = model(**inputs).waveform

The resulting waveform can be saved as a .wav file:

import scipy

scipy.io.wavfile.write("techno.wav", rate=model.config.sampling_rate, data=output)

Or displayed in a Jupyter Notebook / Google Colab:

from IPython.display import Audio

Audio(output, rate=model.config.sampling_rate)

✨ Features

• Multilingual Support: Part of the MMS project, aiming to cover over 1000 languages.
• Stochastic Duration Prediction: Allows for synthesizing speech with different rhythms from the same input text.

📦 Installation

To use this model, you need to install the latest version of the 🤗 Transformers library and the accelerate library:

pip install --upgrade transformers accelerate

💻 Usage Examples

Basic Usage

from transformers import VitsModel, AutoTokenizer
import torch

model = VitsModel.from_pretrained("facebook/mms-tts-guj")
tokenizer = AutoTokenizer.from_pretrained("facebook/mms-tts-guj")

text = "some example text in the Gujarati language"
inputs = tokenizer(text, return_tensors="pt")

with torch.no_grad():
    output = model(**inputs).waveform

Advanced Usage

Save the resulting waveform as a .wav file:

import scipy

scipy.io.wavfile.write("techno.wav", rate=model.config.sampling_rate, data=output)

Display the audio in a Jupyter Notebook / Google Colab:

from IPython.display import Audio

Audio(output, rate=model.config.sampling_rate)

📚 Documentation

Model Details

VITS (Variational Inference with adversarial learning for end-to-end Text-to-Speech) is an end-to-end speech synthesis model. It predicts a speech waveform based on an input text sequence. It's a conditional variational autoencoder (VAE) with a posterior encoder, decoder, and conditional prior.

A flow-based module, consisting of a Transformer-based text encoder and multiple coupling layers, predicts a set of spectrogram-based acoustic features. The spectrogram is decoded using a stack of transposed convolutional layers, similar to the HiFi-GAN vocoder. Given the one-to-many nature of the TTS problem, where the same text can be spoken in multiple ways, the model includes a stochastic duration predictor, enabling it to synthesize speech with different rhythms from the same input text.

The model is trained end-to-end with a combination of losses from variational lower bound and adversarial training. To enhance the model's expressiveness, normalizing flows are applied to the conditional prior distribution. During inference, text encodings are up-sampled based on the duration prediction module and then mapped to the waveform using a cascade of the flow module and HiFi-GAN decoder. Due to the stochastic nature of the duration predictor, the model is non-deterministic and requires a fixed seed to generate the same speech waveform.

For the MMS project, a separate VITS checkpoint is trained for each language.

🔧 Technical Details

The model architecture of VITS is a conditional variational autoencoder (VAE). It consists of a posterior encoder, decoder, and conditional prior. The flow-based module predicts spectrogram-based acoustic features, and the spectrogram is decoded using transposed convolutional layers. The model includes a stochastic duration predictor to handle the one-to-many nature of the TTS problem.

The model is trained end-to-end with a combination of losses from variational lower bound and adversarial training. Normalizing flows are applied to the conditional prior distribution to improve the model's expressiveness. During inference, text encodings are up-sampled based on the duration prediction module and then mapped to the waveform using a cascade of the flow module and HiFi-GAN decoder.

📄 License

The model is licensed under CC-BY-NC 4.0.

BibTex citation

This model was developed by Vineel Pratap et al. from Meta AI. If you use the model, consider citing the MMS paper:

@article{pratap2023mms,
    title={Scaling Speech Technology to 1,000+ Languages},
    author={Vineel Pratap and Andros Tjandra and Bowen Shi and Paden Tomasello and Arun Babu and Sayani Kundu and Ali Elkahky and Zhaoheng Ni and Apoorv Vyas and Maryam Fazel-Zarandi and Alexei Baevski and Yossi Adi and Xiaohui Zhang and Wei-Ning Hsu and Alexis Conneau and Michael Auli},
    journal={arXiv},
    year={2023}
}