Snortts Indic V0

🚀 snorTTS-Indic-v0

snorTTS-Indic-v0 是一個多語言印度語語音合成（TTS）模型，能夠生成九種印度語言的語音，包括印地語、泰米爾語、泰盧固語、馬拉地語、卡納達語、馬拉雅拉姆語、旁遮普語、古吉拉特語和孟加拉語。

👉 閱讀完整博客：訓練最先進的多語言印度語語音合成（TTS）模型 ，瞭解我們是如何構建該模型的。

👉 在我們的演示平臺上試用該模型。

本項目中使用的所有代碼、數據集和模型（包括基礎模型和微調模型）均可供任何人使用和基於其進行二次開發。

🚀 快速開始

你可以通過以下步驟快速開始使用 snorTTS-Indic-v0 模型：

1. 安裝必要的庫（針對Linux系統）：

sudo apt update

sudo apt install -y sox libsox-dev  

2. 使用Python 3.10。
3. 如果你已經安裝了torch，卸載它，讓unsloth來處理：

pip uninstall -y torch torchaudio 

4. 安裝必要的包：

pip install unsloth loguru snac deepfilternet pydub soundfile librosa torchaudio

5. 運行推理代碼：

from unsloth import FastLanguageModel
from snac import SNAC
import soundfile as sf
import numpy as np
from loguru import logger
from df.enhance import init_df, enhance, save_audio
import torch
import librosa
import torchaudio
import os

#Name of the model
MODEL_NAME = 'snorbyte/snorTTS-Indic-v0'
MAX_SEQ_LENGTH = 4096
HUGGINGFACE_TOKEN = "" # ! Add your hugging face token

# Load the model and tokenizer.
model, tokenizer = FastLanguageModel.from_pretrained(
    model_name=MODEL_NAME,
    # load_in_4bit=True,
    max_seq_length=MAX_SEQ_LENGTH,
    token=HUGGINGFACE_TOKEN,
)
logger.success(f"Loaded model: {MODEL_NAME}")


# Load the end of speech token for the tokenizer.
tokeniser_length = 128256
end_of_speech_id = tokeniser_length + 2
pad_token_id = tokeniser_length + 7
audio_start_id = tokeniser_length + 10

pad_token = tokenizer.decode([pad_token_id])
logger.success("Load special tokens for the tokenizer.")

# Wrap Model for Inference
FastLanguageModel.for_inference(model)
logger.success(f"{MODEL_NAME} is ready for inference.")

# Set the padding token and padding side.
tokenizer.pad_token = pad_token
tokenizer.padding_side = "left"
logger.success("Set padding token and padding side for the tokenizer.")

# Load the SNAC model for audio decoding.
snac_model = SNAC.from_pretrained("hubertsiuzdak/snac_24khz")
logger.success("Loaded SNAC model for audio decoding.")

# Load DeepFilter for optional post processing
df_model, df_state, _ = init_df()

# Function to generate audio
def generate_audio(
    row, model, tokenizer, user=False, temperature=0.4, top_p=0.9, repetition_penalty=1.05
):
    try:
        if user:
            prompt = row["eval_text_user"]
        else:
            prompt = row["eval_text_no_user"]
        inputs = tokenizer(prompt, add_special_tokens=False, return_tensors="pt")
        max_tokens = MAX_SEQ_LENGTH - inputs.input_ids.shape[1]
        output = model.generate(
            input_ids=inputs.input_ids.to("cuda"),
            attention_mask=inputs.attention_mask.to("cuda"),
            max_new_tokens=max_tokens,
            temperature=temperature,
            top_p=top_p,
            repetition_penalty=repetition_penalty,
            eos_token_id=end_of_speech_id,
        )
        audio_ids = []
        for id in output[0]:
            if id >= audio_start_id:
                audio_ids.append(id.item())
        clean_audio_ids = []
        for i in range((len(audio_ids) + 1) // 7):
            for j in range(7):
                clean_audio_ids += [audio_ids[7 * i + j] - audio_start_id]
        codes = [[], [], []]
        for i in range((len(clean_audio_ids) + 1) // 7):
            codes[0].append(clean_audio_ids[7 * i])
            codes[1].append(clean_audio_ids[7 * i + 1] - 4096)
            codes[2].append(clean_audio_ids[7 * i + 2] - (2 * 4096))
            codes[2].append(clean_audio_ids[7 * i + 3] - (3 * 4096))
            codes[1].append(clean_audio_ids[7 * i + 4] - (4 * 4096))
            codes[2].append(clean_audio_ids[7 * i + 5] - (5 * 4096))
            codes[2].append(clean_audio_ids[7 * i + 6] - (6 * 4096))
        codes = [
            torch.tensor(codes[0]).unsqueeze(0),
            torch.tensor(codes[1]).unsqueeze(0),
            torch.tensor(codes[2]).unsqueeze(0),
        ]
        try:
            audio = snac_model.decode(codes)
        except Exception as e:
            logger.error(f"Error decoding audio: {e}")
            return None
        return audio.detach().squeeze().to("cpu").numpy()
    except Exception as e:
        logger.error(f"Error decoding audio: {e}")
        return None

# Run inference.
# * Please refer to the training script to create prompt from SNAC tokens.
row = {
    "eval_text_user": f"<custom_token_3><|begin_of_text|>kannada142: ಅಯ್ಯಯ್ಯೋ... Whitefield ಗೆ reach ಆಗೋಕೆ almost 10 hours ಆಯ್ತು you know... traffic was so terrible today <|eot_id|><custom_token_4><custom_token_5><custom_token_1>"
    }

eval_sample = generate_audio(row, model, tokenizer, True)
if eval_sample is None:
    logger.error("Failed to generate audio for evaluation sample.")
else:
    logger.success("Audio Generated. Post Processing Started")

    ## post-processing settings
    filename = "eval.wav"
    speed = 1.05 #add speed up according to speaker
    denoise = False #denoise if you want
    output = eval_sample.astype(np.float32) 

    #speed up
    if abs(speed - 1.0) > 1e-4:
        output_t = torch.from_numpy(output).unsqueeze(0) 
        output_speed, _ = torchaudio.sox_effects.apply_effects_tensor(output_t, 24_000, effects=[["tempo", f"{speed}"]])
        output = output_speed.squeeze(0).cpu().numpy()
    
    #denoise
    if denoise:
        resampled_48k = librosa.resample(output, orig_sr=24_000, target_sr=48_000)
        resampled_48k = torch.from_numpy(resampled_48k).unsqueeze(0)
        output_48k = enhance(df_model, df_state, resampled_48k)
        output_48k = output_48k.squeeze(0).cpu().numpy() 
        output = librosa.resample(output_48k, orig_sr=48_000, target_sr=24_000)
        
    logger.success("Saving Final Output...")
     
    #save
    sf.write(filename, output, 24_000)   

    logger.success(f"Generated and saved evaluation sample audio as {filename}.")

✨ 主要特性

• 語音合成（TTS）
• 語音克隆
• 代碼切換
• 跨語言語音克隆（多語言語音遷移）

📦 安裝指南

安裝必要的庫（針對Linux系統）

sudo apt update

sudo apt install -y sox libsox-dev  

使用Python 3.10

卸載已安裝的torch

pip uninstall -y torch torchaudio 

安裝必要的包

pip install unsloth loguru snac deepfilternet pydub soundfile librosa torchaudio

💻 使用示例

基礎用法

# 運行推理的基礎代碼
from unsloth import FastLanguageModel
from snac import SNAC
import soundfile as sf
import numpy as np
from loguru import logger
from df.enhance import init_df, enhance, save_audio
import torch
import librosa
import torchaudio
import os

#Name of the model
MODEL_NAME = 'snorbyte/snorTTS-Indic-v0'
MAX_SEQ_LENGTH = 4096
HUGGINGFACE_TOKEN = "" # ! Add your hugging face token

# Load the model and tokenizer.
model, tokenizer = FastLanguageModel.from_pretrained(
    model_name=MODEL_NAME,
    # load_in_4bit=True,
    max_seq_length=MAX_SEQ_LENGTH,
    token=HUGGINGFACE_TOKEN,
)
logger.success(f"Loaded model: {MODEL_NAME}")


# Load the end of speech token for the tokenizer.
tokeniser_length = 128256
end_of_speech_id = tokeniser_length + 2
pad_token_id = tokeniser_length + 7
audio_start_id = tokeniser_length + 10

pad_token = tokenizer.decode([pad_token_id])
logger.success("Load special tokens for the tokenizer.")

# Wrap Model for Inference
FastLanguageModel.for_inference(model)
logger.success(f"{MODEL_NAME} is ready for inference.")

# Set the padding token and padding side.
tokenizer.pad_token = pad_token
tokenizer.padding_side = "left"
logger.success("Set padding token and padding side for the tokenizer.")

# Load the SNAC model for audio decoding.
snac_model = SNAC.from_pretrained("hubertsiuzdak/snac_24khz")
logger.success("Loaded SNAC model for audio decoding.")

# Load DeepFilter for optional post processing
df_model, df_state, _ = init_df()

# Function to generate audio
def generate_audio(
    row, model, tokenizer, user=False, temperature=0.4, top_p=0.9, repetition_penalty=1.05
):
    try:
        if user:
            prompt = row["eval_text_user"]
        else:
            prompt = row["eval_text_no_user"]
        inputs = tokenizer(prompt, add_special_tokens=False, return_tensors="pt")
        max_tokens = MAX_SEQ_LENGTH - inputs.input_ids.shape[1]
        output = model.generate(
            input_ids=inputs.input_ids.to("cuda"),
            attention_mask=inputs.attention_mask.to("cuda"),
            max_new_tokens=max_tokens,
            temperature=temperature,
            top_p=top_p,
            repetition_penalty=repetition_penalty,
            eos_token_id=end_of_speech_id,
        )
        audio_ids = []
        for id in output[0]:
            if id >= audio_start_id:
                audio_ids.append(id.item())
        clean_audio_ids = []
        for i in range((len(audio_ids) + 1) // 7):
            for j in range(7):
                clean_audio_ids += [audio_ids[7 * i + j] - audio_start_id]
        codes = [[], [], []]
        for i in range((len(clean_audio_ids) + 1) // 7):
            codes[0].append(clean_audio_ids[7 * i])
            codes[1].append(clean_audio_ids[7 * i + 1] - 4096)
            codes[2].append(clean_audio_ids[7 * i + 2] - (2 * 4096))
            codes[2].append(clean_audio_ids[7 * i + 3] - (3 * 4096))
            codes[1].append(clean_audio_ids[7 * i + 4] - (4 * 4096))
            codes[2].append(clean_audio_ids[7 * i + 5] - (5 * 4096))
            codes[2].append(clean_audio_ids[7 * i + 6] - (6 * 4096))
        codes = [
            torch.tensor(codes[0]).unsqueeze(0),
            torch.tensor(codes[1]).unsqueeze(0),
            torch.tensor(codes[2]).unsqueeze(0),
        ]
        try:
            audio = snac_model.decode(codes)
        except Exception as e:
            logger.error(f"Error decoding audio: {e}")
            return None
        return audio.detach().squeeze().to("cpu").numpy()
    except Exception as e:
        logger.error(f"Error decoding audio: {e}")
        return None

# Run inference.
# * Please refer to the training script to create prompt from SNAC tokens.
row = {
    "eval_text_user": f"<custom_token_3><|begin_of_text|>kannada142: ಅಯ್ಯಯ್ಯೋ... Whitefield ಗೆ reach ಆಗೋಕೆ almost 10 hours ಆಯ್ತು you know... traffic was so terrible today <|eot_id|><custom_token_4><custom_token_5><custom_token_1>"
    }

eval_sample = generate_audio(row, model, tokenizer, True)
if eval_sample is None:
    logger.error("Failed to generate audio for evaluation sample.")
else:
    logger.success("Audio Generated. Post Processing Started")

    ## post-processing settings
    filename = "eval.wav"
    speed = 1.05 #add speed up according to speaker
    denoise = False #denoise if you want
    output = eval_sample.astype(np.float32) 

    #speed up
    if abs(speed - 1.0) > 1e-4:
        output_t = torch.from_numpy(output).unsqueeze(0) 
        output_speed, _ = torchaudio.sox_effects.apply_effects_tensor(output_t, 24_000, effects=[["tempo", f"{speed}"]])
        output = output_speed.squeeze(0).cpu().numpy()
    
    #denoise
    if denoise:
        resampled_48k = librosa.resample(output, orig_sr=24_000, target_sr=48_000)
        resampled_48k = torch.from_numpy(resampled_48k).unsqueeze(0)
        output_48k = enhance(df_model, df_state, resampled_48k)
        output_48k = output_48k.squeeze(0).cpu().numpy() 
        output = librosa.resample(output_48k, orig_sr=48_000, target_sr=24_000)
        
    logger.success("Saving Final Output...")
     
    #save
    sf.write(filename, output, 24_000)   

    logger.success(f"Generated and saved evaluation sample audio as {filename}.")

高級用法

# 可以根據不同的需求調整生成音頻的參數，如溫度、top_p等
# 例如修改 generate_audio 函數中的參數
eval_sample = generate_audio(row, model, tokenizer, True, temperature=0.6, top_p=0.95)

📚 詳細文檔

模型概述

訓練

有關訓練和數據集的詳細信息，請參考 訓練最先進的多語言印度語語音合成（TTS）模型。

你可以在本倉庫中找到訓練腳本 (train_orepheus.py)，它是一個用於微調基礎模型的獨立腳本。

👉 訓練使用的數據集：snorbyte/indic-tts-sample-snac-encoded

提示模板

標準提示

{
    "eval_text_no_user": f"<custom_token_3><|begin_of_text|>{utterance}<|eot_id|><custom_token_4><custom_token_5><custom_token_1>"
}

{
    "eval_text_no_user": f"<custom_token_3><|begin_of_text|>நிச்சயமா. ரோம் ல் இரவு நேரம் ரொம்ப அழகா இருக்கு—piazzaகள் சுத்துறதுக்கு நல்ல நேரம்.<|eot_id|><custom_token_4><custom_token_5><custom_token_1>"
},

特定說話人提示（推薦）

{
    "eval_text_user": f"<custom_token_3><|begin_of_text|>{language}{speaker_id}: {utterance}<|eot_id|><custom_token_4><custom_token_5><custom_token_1>"
}

📝 utterance 可以是說話人的母語、多語言混合或代碼切換的文本。

{
    "eval_text_user": f"<custom_token_3><|begin_of_text|>hindi159:  चलते रहो इस सफर में बिना रुके, क्योंकि मंज़िलें खुद राह दिखाने लगती हैं <|eot_id|><custom_token_4><custom_token_5><custom_token_1>"
}

{
    "eval_text_user": f"<custom_token_3><|begin_of_text|>bengali125: मुझे तो लगा वो आएगा, ஆனா அவன் வந்து full drama பண்ணிட்டான், আর শেষে আবার আমাকে দোষ দিচ্ছে <|eot_id|><custom_token_4><custom_token_5><custom_token_1>"
}

說話人ID

🔧 技術細節

本模型的技術細節可參考 訓練最先進的多語言印度語語音合成（TTS）模型。

📄 許可證

本項目採用 Apache-2.0 許可證。

📞 聯繫我們

👉 郵箱：founders@snorbyte.com

👉 網站：https://snorbyte.com

📚 引用

如果你使用了本模型，請使用以下 BibTeX 引用：

@misc{indictextaudio2025,
  title={snorTTS-Indic-v0: Multilingual Indic TTS},
  author={snorbyte},
  year={2025},
  howpublished={\url{snorbyte/snorTTS-Indic-v0}},
  note={Apache-2.0}
}