Kss Tts Train Jets Raw Phn Null G2pk Train.total Count.ave
模型简介
该模型能够将韩语文本转换为自然语音,适用于语音合成应用。
模型特点
高质量语音合成
采用JETS架构,能够生成自然流畅的韩语语音
端到端训练
整个系统采用端到端方式训练,简化了传统TTS系统的复杂流程
韩语优化
专门针对韩语语音特性进行优化,使用g2pk进行韩语文本处理
模型能力
韩语文本转语音
语音合成
使用案例
语音助手
韩语语音助手
为韩语语音助手提供自然语音输出
有声读物
韩语有声内容生成
将韩语文本内容自动转换为语音
🚀 ESPnet2 TTS模型
本模型是一个基于ESPnet2的文本转语音(TTS)模型,使用kss数据集进行训练,可用于将文本转换为自然流畅的语音。
🚀 快速开始
模型信息
- 模型名称:
imdanboy/kss_tts_train_jets_raw_phn_null_g2pk_train.total_count.ave
- 训练者:satoshi.2020
- 训练方式:使用 espnet 中的kss配方进行训练
演示:在ESPnet2中如何使用
cd espnet
git checkout 047d0c474c18a87c205e566948410be16787e477
pip install -e .
cd egs2/kss/tts1
./run.sh --skip_data_prep false --skip_train true --download_model imdanboy/kss_tts_train_jets_raw_phn_null_g2pk_train.total_count.ave
📚 详细文档
TTS配置
展开
config: conf/tuning/train_jets.yaml
print_config: false
log_level: INFO
dry_run: false
iterator_type: sequence
output_dir: exp/tts_train_jets_raw_phn_null_g2pk
ngpu: 1
seed: 777
num_workers: 4
num_att_plot: 3
dist_backend: nccl
dist_init_method: env://
dist_world_size: 4
dist_rank: 0
local_rank: 0
dist_master_addr: localhost
dist_master_port: 52809
dist_launcher: null
multiprocessing_distributed: true
unused_parameters: true
sharded_ddp: false
cudnn_enabled: true
cudnn_benchmark: false
cudnn_deterministic: false
collect_stats: false
write_collected_feats: false
max_epoch: 1000
patience: null
val_scheduler_criterion:
- valid
- loss
early_stopping_criterion:
- valid
- loss
- min
best_model_criterion:
- - valid
- text2mel_loss
- min
- - train
- text2mel_loss
- min
- - train
- total_count
- max
keep_nbest_models: 5
nbest_averaging_interval: 0
grad_clip: -1
grad_clip_type: 2.0
grad_noise: false
accum_grad: 1
no_forward_run: false
resume: true
train_dtype: float32
use_amp: false
log_interval: 50
use_matplotlib: true
use_tensorboard: true
use_wandb: false
wandb_project: null
wandb_id: null
wandb_entity: null
wandb_name: null
wandb_model_log_interval: -1
detect_anomaly: false
pretrain_path: null
init_param: []
ignore_init_mismatch: false
freeze_param: []
num_iters_per_epoch: 1000
batch_size: 20
valid_batch_size: null
batch_bins: 2000000
valid_batch_bins: null
train_shape_file:
- exp/tts_stats_raw_phn_null_g2pk/train/text_shape.phn
- exp/tts_stats_raw_phn_null_g2pk/train/speech_shape
valid_shape_file:
- exp/tts_stats_raw_phn_null_g2pk/valid/text_shape.phn
- exp/tts_stats_raw_phn_null_g2pk/valid/speech_shape
batch_type: numel
valid_batch_type: null
fold_length:
- 150
- 204800
sort_in_batch: descending
sort_batch: descending
multiple_iterator: false
chunk_length: 500
chunk_shift_ratio: 0.5
num_cache_chunks: 1024
train_data_path_and_name_and_type:
- - dump/raw/tr_no_dev/text
- text
- text
- - dump/raw/tr_no_dev/wav.scp
- speech
- sound
- - exp/tts_stats_raw_phn_null_g2pk/train/collect_feats/pitch.scp
- pitch
- npy
- - exp/tts_stats_raw_phn_null_g2pk/train/collect_feats/energy.scp
- energy
- npy
valid_data_path_and_name_and_type:
- - dump/raw/dev/text
- text
- text
- - dump/raw/dev/wav.scp
- speech
- sound
- - exp/tts_stats_raw_phn_null_g2pk/valid/collect_feats/pitch.scp
- pitch
- npy
- - exp/tts_stats_raw_phn_null_g2pk/valid/collect_feats/energy.scp
- energy
- npy
allow_variable_data_keys: false
max_cache_size: 0.0
max_cache_fd: 32
valid_max_cache_size: null
optim: adamw
optim_conf:
lr: 0.0002
betas:
- 0.8
- 0.99
eps: 1.0e-09
weight_decay: 0.0
scheduler: exponentiallr
scheduler_conf:
gamma: 0.999875
optim2: adamw
optim2_conf:
lr: 0.0002
betas:
- 0.8
- 0.99
eps: 1.0e-09
weight_decay: 0.0
scheduler2: exponentiallr
scheduler2_conf:
gamma: 0.999875
generator_first: true
token_list:
- <blank>
- <unk>
- ''
- ᅡ
- ᅵ
- ᄋ
- ᅳ
- ᄀ
- ᅥ
- ᄂ
- ᆫ
- ᄅ
- ᄌ
- ᄉ
- ᅩ
- ᆯ
- ᄆ
- .
- ᅮ
- ᄃ
- ᄒ
- ᅦ
- ᆼ
- ᅢ
- ᄇ
- ᅭ
- ᅧ
- ᄊ
- ᆷ
- ᄄ
- ᆮ
- ᄎ
- ᄁ
- ᆨ
- ᄑ
- ᄐ
- ᅪ
- ᄏ
- '?'
- ᄍ
- ᆸ
- ᅬ
- ᅣ
- ᅴ
- ᅯ
- ᅨ
- ᄈ
- ᅱ
- ᅲ
- ᅫ
- ','
- '!'
- ᅤ
- ':'
- ᅰ
- ''''
- '-'
- '"'
- /
- I
- M
- F
- E
- S
- C
- A
- B
- ㅇ
- <sos/eos>
odim: null
model_conf: {}
use_preprocessor: true
token_type: phn
bpemodel: null
non_linguistic_symbols: null
cleaner: null
g2p: g2pk
feats_extract: fbank
feats_extract_conf:
n_fft: 1024
hop_length: 256
win_length: null
fs: 24000
fmin: 0
fmax: null
n_mels: 80
normalize: global_mvn
normalize_conf:
stats_file: exp/tts_stats_raw_phn_null_g2pk/train/feats_stats.npz
tts: jets
tts_conf:
generator_type: jets_generator
generator_params:
adim: 256
aheads: 2
elayers: 4
eunits: 1024
dlayers: 4
dunits: 1024
positionwise_layer_type: conv1d
positionwise_conv_kernel_size: 3
duration_predictor_layers: 2
duration_predictor_chans: 256
duration_predictor_kernel_size: 3
use_masking: true
encoder_normalize_before: true
decoder_normalize_before: true
encoder_type: transformer
decoder_type: transformer
conformer_rel_pos_type: latest
conformer_pos_enc_layer_type: rel_pos
conformer_self_attn_layer_type: rel_selfattn
conformer_activation_type: swish
use_macaron_style_in_conformer: true
use_cnn_in_conformer: true
conformer_enc_kernel_size: 7
conformer_dec_kernel_size: 31
init_type: xavier_uniform
transformer_enc_dropout_rate: 0.2
transformer_enc_positional_dropout_rate: 0.2
transformer_enc_attn_dropout_rate: 0.2
transformer_dec_dropout_rate: 0.2
transformer_dec_positional_dropout_rate: 0.2
transformer_dec_attn_dropout_rate: 0.2
pitch_predictor_layers: 5
pitch_predictor_chans: 256
pitch_predictor_kernel_size: 5
pitch_predictor_dropout: 0.5
pitch_embed_kernel_size: 1
pitch_embed_dropout: 0.0
stop_gradient_from_pitch_predictor: true
energy_predictor_layers: 2
energy_predictor_chans: 256
energy_predictor_kernel_size: 3
energy_predictor_dropout: 0.5
energy_embed_kernel_size: 1
energy_embed_dropout: 0.0
stop_gradient_from_energy_predictor: false
generator_out_channels: 1
generator_channels: 512
generator_global_channels: -1
generator_kernel_size: 7
generator_upsample_scales:
- 8
- 8
- 2
- 2
generator_upsample_kernel_sizes:
- 16
- 16
- 4
- 4
generator_resblock_kernel_sizes:
- 3
- 7
- 11
generator_resblock_dilations:
- - 1
- 3
- 5
- - 1
- 3
- 5
- - 1
- 3
- 5
generator_use_additional_convs: true
generator_bias: true
generator_nonlinear_activation: LeakyReLU
generator_nonlinear_activation_params:
negative_slope: 0.1
generator_use_weight_norm: true
segment_size: 64
idim: 69
odim: 80
discriminator_type: hifigan_multi_scale_multi_period_discriminator
discriminator_params:
scales: 1
scale_downsample_pooling: AvgPool1d
scale_downsample_pooling_params:
kernel_size: 4
stride: 2
padding: 2
scale_discriminator_params:
in_channels: 1
out_channels: 1
kernel_sizes:
- 15
- 41
- 5
- 3
channels: 128
max_downsample_channels: 1024
max_groups: 16
bias: true
downsample_scales:
- 2
- 2
- 4
- 4
- 1
nonlinear_activation: LeakyReLU
nonlinear_activation_params:
negative_slope: 0.1
use_weight_norm: true
use_spectral_norm: false
follow_official_norm: false
periods:
- 2
- 3
- 5
- 7
- 11
period_discriminator_params:
in_channels: 1
out_channels: 1
kernel_sizes:
- 5
- 3
channels: 32
downsample_scales:
- 3
- 3
- 3
- 3
- 1
max_downsample_channels: 1024
bias: true
nonlinear_activation: LeakyReLU
nonlinear_activation_params:
negative_slope: 0.1
use_weight_norm: true
use_spectral_norm: false
generator_adv_loss_params:
average_by_discriminators: false
loss_type: mse
discriminator_adv_loss_params:
average_by_discriminators: false
loss_type: mse
feat_match_loss_params:
average_by_discriminators: false
average_by_layers: false
include_final_outputs: true
mel_loss_params:
fs: 24000
n_fft: 1024
hop_length: 256
win_length: null
window: hann
n_mels: 80
fmin: 0
fmax: null
log_base: null
lambda_adv: 1.0
lambda_mel: 45.0
lambda_feat_match: 2.0
lambda_var: 1.0
lambda_align: 2.0
sampling_rate: 24000
cache_generator_outputs: true
pitch_extract: dio
pitch_extract_conf:
reduction_factor: 1
use_token_averaged_f0: false
fs: 24000
n_fft: 1024
hop_length: 256
f0max: 400
f0min: 80
pitch_normalize: global_mvn
pitch_normalize_conf:
stats_file: exp/tts_stats_raw_phn_null_g2pk/train/pitch_stats.npz
energy_extract: energy
energy_extract_conf:
reduction_factor: 1
use_token_averaged_energy: false
fs: 24000
n_fft: 1024
hop_length: 256
win_length: null
energy_normalize: global_mvn
energy_normalize_conf:
stats_file: exp/tts_stats_raw_phn_null_g2pk/train/energy_stats.npz
required:
- output_dir
- token_list
version: '202204'
distributed: true
引用ESPnet
如果您在研究中使用了ESPnet,请引用以下论文:
@inproceedings{watanabe2018espnet,
author={Shinji Watanabe and Takaaki Hori and Shigeki Karita and Tomoki Hayashi and Jiro Nishitoba and Yuya Unno and Nelson Yalta and Jahn Heymann and Matthew Wiesner and Nanxin Chen and Adithya Renduchintala and Tsubasa Ochiai},
title={{ESPnet}: End-to-End Speech Processing Toolkit},
year={2018},
booktitle={Proceedings of Interspeech},
pages={2207--2211},
doi={10.21437/Interspeech.2018-1456},
url={http://dx.doi.org/10.21437/Interspeech.2018-1456}
}
@inproceedings{hayashi2020espnet,
title={{Espnet-TTS}: Unified, reproducible, and integratable open source end-to-end text-to-speech toolkit},
author={Hayashi, Tomoki and Yamamoto, Ryuichi and Inoue, Katsuki and Yoshimura, Takenori and Watanabe, Shinji and Toda, Tomoki and Takeda, Kazuya and Zhang, Yu and Tan, Xu},
booktitle={Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)},
pages={7654--7658},
year={2020},
organization={IEEE}
}
或者引用arXiv上的论文:
@misc{watanabe2018espnet,
title={ESPnet: End-to-End Speech Processing Toolkit},
author={Shinji Watanabe and Takaaki Hori and Shigeki Karita and Tomoki Hayashi and Jiro Nishitoba and Yuya Unno and Nelson Yalta and Jahn Heymann and Matthew Wiesner and Nanxin Chen and Adithya Renduchintala and Tsubasa Ochiai},
year={2018},
eprint={1804.00015},
archivePrefix={arXiv},
primaryClass={cs.CL}
}
📄 许可证
本项目采用CC BY 4.0许可证。
Kokoro 82M
Apache-2.0
Kokoro是一款拥有8200万参数的开源文本转语音(TTS)模型,以其轻量级架构和高音质著称,同时具备快速和成本效益高的特点。
语音合成 英语
K
hexgrad
2.0M
4,155
XTTS V2
其他
ⓍTTS是一款革命性的语音生成模型,仅需6秒音频片段即可实现跨语言音色克隆,支持17种语言。
语音合成
X
coqui
1.7M
2,630
F5 TTS
F5-TTS 是一个基于流匹配的语音合成模型,专注于流畅且忠实的语音合成,特别适用于童话讲述等场景。
语音合成
F
SWivid
851.49k
1,000
Bigvgan V2 22khz 80band 256x
MIT
BigVGAN是基于大规模训练的通用神经声码器,能够从梅尔频谱生成高质量音频波形。
语音合成
B
nvidia
503.23k
16
Speecht5 Tts
MIT
基于LibriTTS数据集微调的SpeechT5语音合成(文本转语音)模型,支持高质量的文本转语音转换。
语音合成
Transformers

S
microsoft
113.83k
760
Dia 1.6B
Apache-2.0
Dia是由Nari实验室开发的16亿参数文本转语音模型,能够直接从文本生成高度逼真的对话,支持情感和语调控制,并能生成非语言交流内容。
语音合成
Safetensors 英语
D
nari-labs
80.28k
1,380
Csm 1b
Apache-2.0
CSM是Sesame开发的10亿参数规模语音生成模型,可根据文本和音频输入生成RVQ音频编码
语音合成
Safetensors 英语
C
sesame
65.03k
1,950
Kokoro 82M V1.1 Zh
Apache-2.0
Kokoro 是一个开放权重的小型但功能强大的文本转语音(TTS)模型系列,新增了来自专业数据集的100名中文说话人数据。
语音合成
K
hexgrad
51.56k
112
Indic Parler Tts
Apache-2.0
Indic Parler-TTS 是 Parler-TTS Mini 的多语言印度语言扩展版本,支持21种语言,包括多种印度语言和英语。
语音合成
Transformers 支持多种语言

I
ai4bharat
43.59k
124
Bark
MIT
Bark是由Suno创建的基于Transformer的文本转音频模型,能生成高度逼真的多语言语音、音乐、背景噪音和简单音效。
语音合成
Transformers 支持多种语言

B
suno
35.72k
1,326
精选推荐AI模型
Llama 3 Typhoon V1.5x 8b Instruct
专为泰语设计的80亿参数指令模型,性能媲美GPT-3.5-turbo,优化了应用场景、检索增强生成、受限生成和推理任务
大型语言模型
Transformers 支持多种语言

L
scb10x
3,269
16
Cadet Tiny
Openrail
Cadet-Tiny是一个基于SODA数据集训练的超小型对话模型,专为边缘设备推理设计,体积仅为Cosmo-3B模型的2%左右。
对话系统
Transformers 英语

C
ToddGoldfarb
2,691
6
Roberta Base Chinese Extractive Qa
基于RoBERTa架构的中文抽取式问答模型,适用于从给定文本中提取答案的任务。
问答系统 中文
R
uer
2,694
98