Mms Tts Ara

🚀 Massively Multilingual Speech (MMS): Arabic Text-to-Speech

This repository provides an Arabic text-to-speech model checkpoint, part of Facebook's Massively Multilingual Speech project aiming to offer speech technology across diverse languages.

🚀 Quick Start

This Arabic text-to-speech (TTS) model is part of Facebook's Massively Multilingual Speech project. You can find more details about supported languages and their ISO 639 - 3 codes in the MMS Language Coverage Overview. All MMS - TTS checkpoints are available on the Hugging Face Hub: facebook/mms-tts.

MMS - TTS is available in the 🤗 Transformers library from version 4.33 onwards.

✨ Features

• Part of the large - scale Massively Multilingual Speech project.
• Based on the VITS end - to - end speech synthesis model.
• Capable of synthesizing Arabic speech from text.

🔧 Technical Details

VITS (Variational Inference with adversarial learning for end - to - end Text - to - Speech) is an end - to - end speech synthesis model. It predicts a speech waveform based on an input text sequence. It is a conditional variational autoencoder (VAE) composed of a posterior encoder, decoder, and conditional prior.

A set of spectrogram - based acoustic features are predicted by the flow - based module, which consists of a Transformer - based text encoder and multiple coupling layers. The spectrogram is decoded using a stack of transposed convolutional layers, similar to the HiFi - GAN vocoder.

Due to the one - to - many nature of the TTS problem (the same text can be spoken in multiple ways), the model includes a stochastic duration predictor, enabling it to synthesize speech with different rhythms from the same input text.

The model is trained end - to - end with a combination of losses from variational lower bound and adversarial training. Normalizing flows are applied to the conditional prior distribution to enhance the model's expressiveness.

During inference, text encodings are up - sampled according to the duration prediction module and then mapped to the waveform using a cascade of the flow module and HiFi - GAN decoder. Because of the stochastic nature of the duration predictor, the model is non - deterministic and requires a fixed seed to generate the same speech waveform.

For the MMS project, a separate VITS checkpoint is trained for each language.

📦 Installation

To use this checkpoint, first install the latest version of the 🤗 Transformers library:

pip install --upgrade transformers accelerate

💻 Usage Examples

Basic Usage

from transformers import VitsModel, AutoTokenizer
import torch

model = VitsModel.from_pretrained("facebook/mms-tts-ara")
tokenizer = AutoTokenizer.from_pretrained("facebook/mms-tts-ara")

text = "some example text in the Arabic language"
inputs = tokenizer(text, return_tensors="pt")

with torch.no_grad():
    output = model(**inputs).waveform

Advanced Usage

Save the resulting waveform as a .wav file:

import scipy

scipy.io.wavfile.write("techno.wav", rate=model.config.sampling_rate, data=output)

Display in a Jupyter Notebook / Google Colab:

from IPython.display import Audio

Audio(output, rate=model.config.sampling_rate)

📄 License

The model is licensed as CC - BY - NC 4.0.

BibTex citation

This model was developed by Vineel Pratap et al. from Meta AI. If you use the model, consider citing the MMS paper:

@article{pratap2023mms,
    title={Scaling Speech Technology to 1,000+ Languages},
    author={Vineel Pratap and Andros Tjandra and Bowen Shi and Paden Tomasello and Arun Babu and Sayani Kundu and Ali Elkahky and Zhaoheng Ni and Apoorv Vyas and Maryam Fazel - Zarandi and Alexei Baevski and Yossi Adi and Xiaohui Zhang and Wei - Ning Hsu and Alexis Conneau and Michael Auli},
    journal={arXiv},
    year={2023}
}