Speecht5 Tts Finetuned Nst Da

🚀 speecht5_tts-finetuned-nst-da

This model is a fine-tuned version of microsoft/speecht5_tts on the NST Danish ASR Database dataset. It aims to provide a high - quality solution for Danish text - to - speech synthesis, with reasonable output quality and inference time.

🚀 Quick Start

This model is designed for Danish text - to - speech synthesis. An example script showing how to use the model for inference can be found here.

✨ Features

• Alternative for Danish TTS: Given that Danish is a low - resource language, there are not many open - source Danish text - to - speech synthesizers. This model provides a simpler alternative with good performance in output quality and inference time.
• Easy - to - use Interface: It has an associated Space on 🤗 at JackismyShephard/danish-speech-synthesis for easy Danish text - to - speech synthesis and optional speech enhancement.

📦 Installation

The README does not provide specific installation steps, so this section is skipped.

💻 Usage Examples

The README does not provide code examples, so this section is skipped.

📚 Documentation

Model Description

Given that Danish is a low - resource language, not many open - source implementations of a Danish text - to - speech synthesizer are available online. As of writing, the only other existing implementations available on 🤗 are facebook/seamless-streaming and audo/seamless-m4t-v2-large. This model has been developed to provide a simpler alternative that still performs reasonably well, both in terms of output quality and inference time. Additionally, contrary to the aforementioned models, this model also has an associated Space on 🤗 at JackismyShephard/danish-speech-synthesis which provides an easy interface for Danish text - to - speech synthesis, as well as optional speech enhancement.

Intended Uses & Limitations

• Intended Use: The model is intended for Danish text - to - speech synthesis.
• Limitations:
  • The model does not recognize special symbols such as "æ", "ø" and "å", as it uses the default tokenizer of microsoft/speecht5_tts.
  • The model performs best for short - to - medium - length input text and expects input text to contain no more than 600 vocabulary tokens.
  • For best performance, the model should be given a Danish speaker embedding, ideally generated from an audio clip from the training split of alexandrainst/nst-da using speechbrain/spkrec-xvect-voxceleb.
  • The output of the model is a log - mel spectogram, which should be converted to a waveform using microsoft/speecht5_hifigan. For better quality output, the resulting waveform can be enhanced using ResembleAI/resemble-enhance.

Training and Evaluation Data

The model was trained and evaluated on alexandrainst/nst-da using MSE as both loss and metric. The dataset was pre - processed as follows:

• Special characters, such as "æ", "ø" and "å" were translated to their latin equivalents and examples with text containing digits were removed, as neither are in the vocabulary of the tokenizer of microsoft/speecht5_tts.
• Training split balancing was done by excluding speakers with less than 280 examples or more than 327 examples.
• Audio was enhanced using speechbrain/metricgan-plus-voicebank in an attempt to remove unwanted noise.

Training Procedure

The script used for training the model (and pre - processing its data) can be found here.

Training Hyperparameters

The following hyperparameters were used during training:

• learning_rate: 1e - 05
• train_batch_size: 16
• eval_batch_size: 16
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon = 1e - 08
• lr_scheduler_type: linear
• lr_scheduler_warmup_ratio: 0.1
• num_epochs: 20
• mixed_precision_training: Native AMP

Training Results

Framework Versions

• Transformers 4.37.2
• Pytorch 2.1.1+cu121
• Datasets 2.17.0
• Tokenizers 0.15.2

🔧 Technical Details

The README does not provide specific technical details (more than 50 - word technical descriptions), so this section is skipped.

📄 License

The model is released under the MIT license.