Canary Tts 0.5b

🚀 Canary-TTS-0.5B

A TTS model trained based on sbintuitions/sarashina2.2-0.5b-instruct-v0.1. It adopts the same prompt method as Parler-TTS, allowing fine-grained control of voice quality by changing control prompts and reading prompts.

📚 Canary-TTS Index

• Canary-TTS 0.5B
• Canary-TTS 150M
• Canary-TTS Nano 150M β
• Canary-TTS Nano 150M β Just Sample

🚀 Quick Start

📋 Quick Index

• Model
• Features
• Installation
• Usage
  • Random Voice Generation
  • Speaker Specification
• Sample Audios
• Acknowledgements
• License

📦 Model

✨ Features

• Control of pitch, gender, and noise through control prompts.
• Text-to-speech based on reading prompts.
• Built on the codebase of Parler-TTS and XCodec2.
• Based on llama, enabling the transfer of LLM technologies.

📥 Installation

pip install torch torchvision torchaudio
pip install git+https://github.com/getuka/canary-tts.git

💻 Usage

🔀 Random Voice Generation

import torch, torchaudio
from transformers import AutoModelForCausalLM, AutoTokenizer
from canary_tts.xcodec2.modeling_xcodec2 import XCodec2Model
from rubyinserter import add_ruby

tokenizer = AutoTokenizer.from_pretrained("2121-8/canary-tts-0.5b")
model = AutoModelForCausalLM.from_pretrained("2121-8/canary-tts-0.5b", device_map="auto", torch_dtype=torch.bfloat16)
codec = XCodec2Model.from_pretrained("HKUSTAudio/xcodec2")

description = "A man voice, with a very hight pitch, speaks in a monotone manner. The recording quality is very noises and close-sounding, indicating a good or excellent audio capture."
prompt = 'こんにちは。お元気ですか？'

prompt = add_ruby(prompt)
chat = [
    {"role": "system", "content": description},
    {"role": "user", "content": prompt}
]
tokenized_input = tokenizer.apply_chat_template(chat, add_generation_prompt=True, tokenize=True, return_tensors="pt").to(model.device)

with torch.no_grad():
    output = model.generate(
        tokenized_input,
        max_new_tokens=256,
        top_p=0.95,
        temperature=0.7,
        repetition_penalty=1.05,
    )[0]

audio_tokens = output[len(tokenized_input[0]):]
output_audios = codec.decode_code(audio_tokens.unsqueeze(0).unsqueeze(0).cpu())
torchaudio.save("sample.wav", src=output_audios[0].cpu(), sample_rate=16000)

🎵 Sample Audios

🙏 Acknowledgements

• The Parler-TTS community.
• The developers of XCodec2.

📄 License

CC BY-NC 4.0

📋 Credit

Audio Decoder

• Repository: HKUSTAudio/xcodec2
• License: CC BY-NC 4.0

Model

• Repository: sbintuitions/sarashina2.2-0.5b-instruct-v0.1
• License: MIT License

Copyright and Disclaimer

Please comply with the following conditions:

1. Disclaimer of Appropriateness The creator makes no warranties regarding the accuracy, legality, or appropriateness of the results obtained from using this model.

2. User's Responsibility When using this model, users must comply with all applicable laws and regulations. All responsibilities arising from the generated content shall be borne by the user.

3. Creator's Disclaimer The creator of this repository and the model shall not be liable for any copyright infringement or other legal issues.

4. Response to Deletion Requests In case of copyright issues, the problematic resources or data will be promptly deleted.