Indri 0.1 350m Tts

🚀 Indri-0.1-350m-tts

Indri is a series of audio models capable of performing TTS, ASR, and audio continuation. This medium-sized model (350M) in the series supports TTS tasks in English and Hindi, offering high - quality audio generation.

Key Information

Useful Links

🚀 Quick Start

🤗 pipelines

Use the following code to start using the model. Pipelines are the optimal way to get started.

import torch
import torchaudio
from transformers import pipeline

model_id = '11mlabs/indri-0.1-350m-tts'
task = 'indri-tts'

pipe = pipeline(
    task,
    model=model_id,
    device=torch.device('cuda:0'), # Update this based on your hardware,
    trust_remote_code=True
)

output = pipe(['Hi, my name is Indri and I like to talk.'], speaker = '[spkr_63]')

torchaudio.save('output.wav', output[0]['audio'][0], sample_rate=24000)

Available speakers

Self hosted service

git clone https://github.com/cmeraki/indri.git
cd indri
pip install -r requirements.txt

# Install ffmpeg (for Mac/Windows, refer here: https://www.ffmpeg.org/download.html)
sudo apt update -y
sudo apt upgrade -y
sudo apt install ffmpeg -y

python -m inference --model_path 11mlabs/indri-0.1-350m-tts --device cuda:0 --port 8000

✨ Features

1. Small, based on GPT - 2 medium architecture. The methodology can be extended to any autoregressive transformer - based architecture.
2. Ultra - fast. Using our self hosted service option, on RTX6000Ada NVIDIA GPU the model can achieve speeds up to 300 toks/s (3s of audio generation per s) and under 20ms time to first token.
3. On RTX6000Ada, it can support a batch size of ~300 sequences with full context length of 1024 tokens.
4. Supports voice cloning with small prompts (<5s).
5. Code mixing text input in 2 languages - English and Hindi.

📚 Documentation

Model Details

Model Description

indri-0.1-350m-tts is a novel, ultra - small, and lightweight TTS model based on the transformer architecture. It models audio as tokens and can generate high - quality audio with consistent style cloning of the speaker.

Samples

Details

1. Model Type: GPT - 2 based language model
2. Size: 350M parameters
3. Language Support: English, Hindi
4. License: This model is not for commercial usage. This is only a research showcase.

🔧 Technical Details

Here's a brief of how the model works:

1. Converts input text into tokens.
2. Runs autoregressive decoding on GPT - 2 based transformer model and generates audio tokens.
3. Decodes audio tokens (using Kyutai/mimi) to audio.

Please read our blog here for more technical details on how it was built.

📄 License

This model is released under the cc - by - sa - 4.0 license and is not for commercial usage. It is only a research showcase.

Citation

If you use this model in your research, please cite:

@misc{indri-multimodal-alm,
  author       = {11mlabs},
  title        = {Indri: Multimodal audio language model},
  year         = {2024},
  publisher    = {GitHub},
  journal      = {GitHub Repository},
  howpublished = {\url{https://github.com/cmeraki/indri}},
  email        = {compute@merakilabs.com}
}

BibTex

1. nanoGPT

@techreport{kyutai2024moshi,
      title={Moshi: a speech-text foundation model for real-time dialogue},
      author={Alexandre D\'efossez and Laurent Mazar\'e and Manu Orsini and
      Am\'elie Royer and Patrick P\'erez and Herv\'e J\'egou and Edouard Grave and Neil Zeghidour},
      year={2024},
      eprint={2410.00037},
      archivePrefix={arXiv},
      primaryClass={eess.AS},
      url={https://arxiv.org/abs/2410.00037},
}

2. Whisper

@misc{radford2022whisper,
  doi = {10.48550/ARXIV.2212.04356},
  url = {https://arxiv.org/abs/2212.04356},
  author = {Radford, Alec and Kim, Jong Wook and Xu, Tao and Brockman, Greg and McLeavey, Christine and Sutskever, Ilya},
  title = {Robust Speech Recognition via Large-Scale Weak Supervision},
  publisher = {arXiv},
  year = {2022},
  copyright = {arXiv.org perpetual, non-exclusive license}
}

3. silero-vad

@misc{Silero VAD,
  author = {Silero Team},
  title = {Silero VAD: pre-trained enterprise-grade Voice Activity Detector (VAD), Number Detector and Language Classifier},
  year = {2024},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {\url{https://github.com/snakers4/silero-vad}},
  commit = {insert_some_commit_here},
  email = {hello@silero.ai}
}