Vits Cmn

🚀 VITS: Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech

VITS is an end-to-end speech synthesis model that generates speech waveforms based on input text sequences. It uses a conditional variational autoencoder (VAE) with a posterior encoder, decoder, and conditional prior.

🚀 Quick Start

VITS is an end-to-end speech synthesis model that predicts a speech waveform conditional on an input text sequence. It is a conditional variational autoencoder (VAE) comprised of a posterior encoder, decoder, and conditional prior.

📚 Documentation

Model Details

💻 Usage Examples

Basic Usage

# Using this checkpoint from Hugging Face Transformers
from transformers import VitsModel, VitsTokenizer
from pypinyin import lazy_pinyin, Style
import torch

model = VitsModel.from_pretrained("BricksDisplay/vits-cmn")
tokenizer = VitsTokenizer.from_pretrained("BricksDisplay/vits-cmn")

text = "中文"
payload = ''.join(lazy_pinyin(text, style=Style.TONE, tone_sandhi=True))
inputs = tokenizer(payload, return_tensors="pt")

with torch.no_grad():
    output = model(**inputs, speaker_id=0)

from IPython.display import Audio
Audio(output.audio[0], rate=16000)

Advanced Usage

// Using this checkpoint from Transformers.js
import { pipeline } from '@xenova/transformers';
import { pinyin } from 'pinyin-pro'; // Our use-case, using `pinyin-pro`

const synthesizer = await pipeline('text-to-audio', 'BricksDisplay/vits-cmn', { quantized: false })
console.log(await synthesizer(pinyin("中文")))
// {
//   audio: Float32Array(?) [ ... ],
//   sampling_rate: 16000
// }

⚠️ Important Note

Transformers.js (ONNX) version does not support speaker_id, so it will fixed in 0

📄 License

This project is licensed under the Apache-2.0 license.