Snortts Indic V0

🚀 snorTTS-Indic-v0

snorTTS-Indic-v0 是一个多语言印度语语音合成（TTS）模型，能够生成九种印度语言的语音，包括印地语、泰米尔语、泰卢固语、马拉地语、卡纳达语、马拉雅拉姆语、旁遮普语、古吉拉特语和孟加拉语。

👉 阅读完整博客：训练最先进的多语言印度语语音合成（TTS）模型 ，了解我们是如何构建该模型的。

👉 在我们的演示平台上试用该模型。

本项目中使用的所有代码、数据集和模型（包括基础模型和微调模型）均可供任何人使用和基于其进行二次开发。

🚀 快速开始

你可以通过以下步骤快速开始使用 snorTTS-Indic-v0 模型：

1. 安装必要的库（针对Linux系统）：

sudo apt update

sudo apt install -y sox libsox-dev  

2. 使用Python 3.10。
3. 如果你已经安装了torch，卸载它，让unsloth来处理：

pip uninstall -y torch torchaudio 

4. 安装必要的包：

pip install unsloth loguru snac deepfilternet pydub soundfile librosa torchaudio

5. 运行推理代码：

from unsloth import FastLanguageModel
from snac import SNAC
import soundfile as sf
import numpy as np
from loguru import logger
from df.enhance import init_df, enhance, save_audio
import torch
import librosa
import torchaudio
import os

#Name of the model
MODEL_NAME = 'snorbyte/snorTTS-Indic-v0'
MAX_SEQ_LENGTH = 4096
HUGGINGFACE_TOKEN = "" # ! Add your hugging face token

# Load the model and tokenizer.
model, tokenizer = FastLanguageModel.from_pretrained(
    model_name=MODEL_NAME,
    # load_in_4bit=True,
    max_seq_length=MAX_SEQ_LENGTH,
    token=HUGGINGFACE_TOKEN,
)
logger.success(f"Loaded model: {MODEL_NAME}")


# Load the end of speech token for the tokenizer.
tokeniser_length = 128256
end_of_speech_id = tokeniser_length + 2
pad_token_id = tokeniser_length + 7
audio_start_id = tokeniser_length + 10

pad_token = tokenizer.decode([pad_token_id])
logger.success("Load special tokens for the tokenizer.")

# Wrap Model for Inference
FastLanguageModel.for_inference(model)
logger.success(f"{MODEL_NAME} is ready for inference.")

# Set the padding token and padding side.
tokenizer.pad_token = pad_token
tokenizer.padding_side = "left"
logger.success("Set padding token and padding side for the tokenizer.")

# Load the SNAC model for audio decoding.
snac_model = SNAC.from_pretrained("hubertsiuzdak/snac_24khz")
logger.success("Loaded SNAC model for audio decoding.")

# Load DeepFilter for optional post processing
df_model, df_state, _ = init_df()

# Function to generate audio
def generate_audio(
    row, model, tokenizer, user=False, temperature=0.4, top_p=0.9, repetition_penalty=1.05
):
    try:
        if user:
            prompt = row["eval_text_user"]
        else:
            prompt = row["eval_text_no_user"]
        inputs = tokenizer(prompt, add_special_tokens=False, return_tensors="pt")
        max_tokens = MAX_SEQ_LENGTH - inputs.input_ids.shape[1]
        output = model.generate(
            input_ids=inputs.input_ids.to("cuda"),
            attention_mask=inputs.attention_mask.to("cuda"),
            max_new_tokens=max_tokens,
            temperature=temperature,
            top_p=top_p,
            repetition_penalty=repetition_penalty,
            eos_token_id=end_of_speech_id,
        )
        audio_ids = []
        for id in output[0]:
            if id >= audio_start_id:
                audio_ids.append(id.item())
        clean_audio_ids = []
        for i in range((len(audio_ids) + 1) // 7):
            for j in range(7):
                clean_audio_ids += [audio_ids[7 * i + j] - audio_start_id]
        codes = [[], [], []]
        for i in range((len(clean_audio_ids) + 1) // 7):
            codes[0].append(clean_audio_ids[7 * i])
            codes[1].append(clean_audio_ids[7 * i + 1] - 4096)
            codes[2].append(clean_audio_ids[7 * i + 2] - (2 * 4096))
            codes[2].append(clean_audio_ids[7 * i + 3] - (3 * 4096))
            codes[1].append(clean_audio_ids[7 * i + 4] - (4 * 4096))
            codes[2].append(clean_audio_ids[7 * i + 5] - (5 * 4096))
            codes[2].append(clean_audio_ids[7 * i + 6] - (6 * 4096))
        codes = [
            torch.tensor(codes[0]).unsqueeze(0),
            torch.tensor(codes[1]).unsqueeze(0),
            torch.tensor(codes[2]).unsqueeze(0),
        ]
        try:
            audio = snac_model.decode(codes)
        except Exception as e:
            logger.error(f"Error decoding audio: {e}")
            return None
        return audio.detach().squeeze().to("cpu").numpy()
    except Exception as e:
        logger.error(f"Error decoding audio: {e}")
        return None

# Run inference.
# * Please refer to the training script to create prompt from SNAC tokens.
row = {
    "eval_text_user": f"<custom_token_3><|begin_of_text|>kannada142: ಅಯ್ಯಯ್ಯೋ... Whitefield ಗೆ reach ಆಗೋಕೆ almost 10 hours ಆಯ್ತು you know... traffic was so terrible today <|eot_id|><custom_token_4><custom_token_5><custom_token_1>"
    }

eval_sample = generate_audio(row, model, tokenizer, True)
if eval_sample is None:
    logger.error("Failed to generate audio for evaluation sample.")
else:
    logger.success("Audio Generated. Post Processing Started")

    ## post-processing settings
    filename = "eval.wav"
    speed = 1.05 #add speed up according to speaker
    denoise = False #denoise if you want
    output = eval_sample.astype(np.float32) 

    #speed up
    if abs(speed - 1.0) > 1e-4:
        output_t = torch.from_numpy(output).unsqueeze(0) 
        output_speed, _ = torchaudio.sox_effects.apply_effects_tensor(output_t, 24_000, effects=[["tempo", f"{speed}"]])
        output = output_speed.squeeze(0).cpu().numpy()
    
    #denoise
    if denoise:
        resampled_48k = librosa.resample(output, orig_sr=24_000, target_sr=48_000)
        resampled_48k = torch.from_numpy(resampled_48k).unsqueeze(0)
        output_48k = enhance(df_model, df_state, resampled_48k)
        output_48k = output_48k.squeeze(0).cpu().numpy() 
        output = librosa.resample(output_48k, orig_sr=48_000, target_sr=24_000)
        
    logger.success("Saving Final Output...")
     
    #save
    sf.write(filename, output, 24_000)   

    logger.success(f"Generated and saved evaluation sample audio as {filename}.")

✨ 主要特性

• 语音合成（TTS）
• 语音克隆
• 代码切换
• 跨语言语音克隆（多语言语音迁移）

📦 安装指南

安装必要的库（针对Linux系统）

sudo apt update

sudo apt install -y sox libsox-dev  

使用Python 3.10

卸载已安装的torch

pip uninstall -y torch torchaudio 

安装必要的包

pip install unsloth loguru snac deepfilternet pydub soundfile librosa torchaudio

💻 使用示例

基础用法

# 运行推理的基础代码
from unsloth import FastLanguageModel
from snac import SNAC
import soundfile as sf
import numpy as np
from loguru import logger
from df.enhance import init_df, enhance, save_audio
import torch
import librosa
import torchaudio
import os

#Name of the model
MODEL_NAME = 'snorbyte/snorTTS-Indic-v0'
MAX_SEQ_LENGTH = 4096
HUGGINGFACE_TOKEN = "" # ! Add your hugging face token

# Load the model and tokenizer.
model, tokenizer = FastLanguageModel.from_pretrained(
    model_name=MODEL_NAME,
    # load_in_4bit=True,
    max_seq_length=MAX_SEQ_LENGTH,
    token=HUGGINGFACE_TOKEN,
)
logger.success(f"Loaded model: {MODEL_NAME}")


# Load the end of speech token for the tokenizer.
tokeniser_length = 128256
end_of_speech_id = tokeniser_length + 2
pad_token_id = tokeniser_length + 7
audio_start_id = tokeniser_length + 10

pad_token = tokenizer.decode([pad_token_id])
logger.success("Load special tokens for the tokenizer.")

# Wrap Model for Inference
FastLanguageModel.for_inference(model)
logger.success(f"{MODEL_NAME} is ready for inference.")

# Set the padding token and padding side.
tokenizer.pad_token = pad_token
tokenizer.padding_side = "left"
logger.success("Set padding token and padding side for the tokenizer.")

# Load the SNAC model for audio decoding.
snac_model = SNAC.from_pretrained("hubertsiuzdak/snac_24khz")
logger.success("Loaded SNAC model for audio decoding.")

# Load DeepFilter for optional post processing
df_model, df_state, _ = init_df()

# Function to generate audio
def generate_audio(
    row, model, tokenizer, user=False, temperature=0.4, top_p=0.9, repetition_penalty=1.05
):
    try:
        if user:
            prompt = row["eval_text_user"]
        else:
            prompt = row["eval_text_no_user"]
        inputs = tokenizer(prompt, add_special_tokens=False, return_tensors="pt")
        max_tokens = MAX_SEQ_LENGTH - inputs.input_ids.shape[1]
        output = model.generate(
            input_ids=inputs.input_ids.to("cuda"),
            attention_mask=inputs.attention_mask.to("cuda"),
            max_new_tokens=max_tokens,
            temperature=temperature,
            top_p=top_p,
            repetition_penalty=repetition_penalty,
            eos_token_id=end_of_speech_id,
        )
        audio_ids = []
        for id in output[0]:
            if id >= audio_start_id:
                audio_ids.append(id.item())
        clean_audio_ids = []
        for i in range((len(audio_ids) + 1) // 7):
            for j in range(7):
                clean_audio_ids += [audio_ids[7 * i + j] - audio_start_id]
        codes = [[], [], []]
        for i in range((len(clean_audio_ids) + 1) // 7):
            codes[0].append(clean_audio_ids[7 * i])
            codes[1].append(clean_audio_ids[7 * i + 1] - 4096)
            codes[2].append(clean_audio_ids[7 * i + 2] - (2 * 4096))
            codes[2].append(clean_audio_ids[7 * i + 3] - (3 * 4096))
            codes[1].append(clean_audio_ids[7 * i + 4] - (4 * 4096))
            codes[2].append(clean_audio_ids[7 * i + 5] - (5 * 4096))
            codes[2].append(clean_audio_ids[7 * i + 6] - (6 * 4096))
        codes = [
            torch.tensor(codes[0]).unsqueeze(0),
            torch.tensor(codes[1]).unsqueeze(0),
            torch.tensor(codes[2]).unsqueeze(0),
        ]
        try:
            audio = snac_model.decode(codes)
        except Exception as e:
            logger.error(f"Error decoding audio: {e}")
            return None
        return audio.detach().squeeze().to("cpu").numpy()
    except Exception as e:
        logger.error(f"Error decoding audio: {e}")
        return None

# Run inference.
# * Please refer to the training script to create prompt from SNAC tokens.
row = {
    "eval_text_user": f"<custom_token_3><|begin_of_text|>kannada142: ಅಯ್ಯಯ್ಯೋ... Whitefield ಗೆ reach ಆಗೋಕೆ almost 10 hours ಆಯ್ತು you know... traffic was so terrible today <|eot_id|><custom_token_4><custom_token_5><custom_token_1>"
    }

eval_sample = generate_audio(row, model, tokenizer, True)
if eval_sample is None:
    logger.error("Failed to generate audio for evaluation sample.")
else:
    logger.success("Audio Generated. Post Processing Started")

    ## post-processing settings
    filename = "eval.wav"
    speed = 1.05 #add speed up according to speaker
    denoise = False #denoise if you want
    output = eval_sample.astype(np.float32) 

    #speed up
    if abs(speed - 1.0) > 1e-4:
        output_t = torch.from_numpy(output).unsqueeze(0) 
        output_speed, _ = torchaudio.sox_effects.apply_effects_tensor(output_t, 24_000, effects=[["tempo", f"{speed}"]])
        output = output_speed.squeeze(0).cpu().numpy()
    
    #denoise
    if denoise:
        resampled_48k = librosa.resample(output, orig_sr=24_000, target_sr=48_000)
        resampled_48k = torch.from_numpy(resampled_48k).unsqueeze(0)
        output_48k = enhance(df_model, df_state, resampled_48k)
        output_48k = output_48k.squeeze(0).cpu().numpy() 
        output = librosa.resample(output_48k, orig_sr=48_000, target_sr=24_000)
        
    logger.success("Saving Final Output...")
     
    #save
    sf.write(filename, output, 24_000)   

    logger.success(f"Generated and saved evaluation sample audio as {filename}.")

高级用法

# 可以根据不同的需求调整生成音频的参数，如温度、top_p等
# 例如修改 generate_audio 函数中的参数
eval_sample = generate_audio(row, model, tokenizer, True, temperature=0.6, top_p=0.95)

📚 详细文档

模型概述

训练

有关训练和数据集的详细信息，请参考 训练最先进的多语言印度语语音合成（TTS）模型。

你可以在本仓库中找到训练脚本 (train_orepheus.py)，它是一个用于微调基础模型的独立脚本。

👉 训练使用的数据集：snorbyte/indic-tts-sample-snac-encoded

提示模板

标准提示

{
    "eval_text_no_user": f"<custom_token_3><|begin_of_text|>{utterance}<|eot_id|><custom_token_4><custom_token_5><custom_token_1>"
}

{
    "eval_text_no_user": f"<custom_token_3><|begin_of_text|>நிச்சயமா. ரோம் ல் இரவு நேரம் ரொம்ப அழகா இருக்கு—piazzaகள் சுத்துறதுக்கு நல்ல நேரம்.<|eot_id|><custom_token_4><custom_token_5><custom_token_1>"
},

特定说话人提示（推荐）

{
    "eval_text_user": f"<custom_token_3><|begin_of_text|>{language}{speaker_id}: {utterance}<|eot_id|><custom_token_4><custom_token_5><custom_token_1>"
}

📝 utterance 可以是说话人的母语、多语言混合或代码切换的文本。

{
    "eval_text_user": f"<custom_token_3><|begin_of_text|>hindi159:  चलते रहो इस सफर में बिना रुके, क्योंकि मंज़िलें खुद राह दिखाने लगती हैं <|eot_id|><custom_token_4><custom_token_5><custom_token_1>"
}

{
    "eval_text_user": f"<custom_token_3><|begin_of_text|>bengali125: मुझे तो लगा वो आएगा, ஆனா அவன் வந்து full drama பண்ணிட்டான், আর শেষে আবার আমাকে দোষ দিচ্ছে <|eot_id|><custom_token_4><custom_token_5><custom_token_1>"
}

说话人ID

🔧 技术细节

本模型的技术细节可参考 训练最先进的多语言印度语语音合成（TTS）模型。

📄 许可证

本项目采用 Apache-2.0 许可证。

📞 联系我们

👉 邮箱：founders@snorbyte.com

👉 网站：https://snorbyte.com

📚 引用

如果你使用了本模型，请使用以下 BibTeX 引用：

@misc{indictextaudio2025,
  title={snorTTS-Indic-v0: Multilingual Indic TTS},
  author={snorbyte},
  year={2025},
  howpublished={\url{snorbyte/snorTTS-Indic-v0}},
  note={Apache-2.0}
}