Indic Seamless

🚀 IndicSeamless語音轉文本翻譯模型

IndicSeamless是一個用於印度語言語音轉文本翻譯的模型，它基於SeamlessM4T - v2在特定數據集上微調而來，在性能上超越了基礎模型和其他競爭系統。

🚀 快速開始

安裝

確保你已經安裝了所需的依賴項：

pip install torch torchaudio transformers datasets

加載模型

import torchaudio
from transformers import SeamlessM4Tv2ForSpeechToText
from transformers import SeamlessM4TTokenizer, SeamlessM4TFeatureExtractor

model = SeamlessM4Tv2ForSpeechToText.from_pretrained("ai4bharat/indic-seamless").to("cuda")
processor = SeamlessM4TFeatureExtractor.from_pretrained("ai4bharat/indic-seamless")
tokenizer = SeamlessM4TTokenizer.from_pretrained("ai4bharat/indic-seamless")

單音頻推理

audio, orig_freq = torchaudio.load("../10002398547238927970.wav")
audio = torchaudio.functional.resample(audio, orig_freq=orig_freq, new_freq=16_000) # must be a 16 kHz waveform array
audio_inputs = processor(audio, sampling_rate=16_000, return_tensors="pt").to("cuda")

text_out = model.generate(**audio_inputs, tgt_lang="hin")[0].cpu().numpy().squeeze()
print(tokenizer.decode(text_out, clean_up_tokenization_spaces=True, skip_special_tokens=True))

在Fleurs數據集上推理

from datasets import load_dataset

dataset = load_dataset("google/fleurs", "hi_in", split="test")

def process_audio(example):
    audio = example["audio"]["array"]
    audio_inputs = processor(audio, sampling_rate=16_000, return_tensors="pt").to("cuda")
    text_out = model.generate(**audio_inputs, tgt_lang="hin")[0].cpu().numpy().squeeze()
    return {"predicted_text": tokenizer.decode(text_out, clean_up_tokenization_spaces=True, skip_special_tokens=True)}

dataset = dataset.map(process_audio)
dataset = dataset.remove_columns(["audio"])
dataset.to_csv("fleurs_hi_predictions.csv")

使用Fleurs進行批量翻譯

from datasets import load_dataset
import torch

def process_batch(batch):
    audio_arrays = [audio["array"] for audio in batch["audio"]]
    audio_inputs = processor(audio_arrays, sampling_rate=16_000, return_tensors="pt", padding=True).to("cuda")
    text_outs = model.generate(**audio_inputs, tgt_lang="hin")
    batch["predicted_text"] = [tokenizer.decode(text_out.cpu().numpy().squeeze(), clean_up_tokenization_spaces=True, skip_special_tokens=True) for text_out in text_outs]
    return batch

def batch_translate(language_code="hi_in", tgt_lang="hin"):
    dataset = load_dataset("google/fleurs", language_code, split="test")
    dataset = dataset.map(process_batch, batched=True, batch_size=8)
    return dataset["predicted_text"]

# Example usage
target_language = "hi_in"
translations = batch_translate(target_language, tgt_lang="hin")
print(translations)

✨ 主要特性

• 性能卓越：該模型在性能上超越了基礎的SeamlessM4Tv2模型以及所有競爭的語音轉文本系統，包括級聯方法。
• 創造新紀錄：在Fleurs數據集上取得了新的最優成績，並在BhasaAnuvaad測試集上顯著超越了所有其他系統，該測試集包含來自新領域的多樣化數據。

📦 安裝指南

確保你已經安裝了所需的依賴項：

pip install torch torchaudio transformers datasets

💻 使用示例

基礎用法

import torchaudio
from transformers import SeamlessM4Tv2ForSpeechToText
from transformers import SeamlessM4TTokenizer, SeamlessM4TFeatureExtractor

model = SeamlessM4Tv2ForSpeechToText.from_pretrained("ai4bharat/indic-seamless").to("cuda")
processor = SeamlessM4TFeatureExtractor.from_pretrained("ai4bharat/indic-seamless")
tokenizer = SeamlessM4TTokenizer.from_pretrained("ai4bharat/indic-seamless")

audio, orig_freq = torchaudio.load("../10002398547238927970.wav")
audio = torchaudio.functional.resample(audio, orig_freq=orig_freq, new_freq=16_000) # must be a 16 kHz waveform array
audio_inputs = processor(audio, sampling_rate=16_000, return_tensors="pt").to("cuda")

text_out = model.generate(**audio_inputs, tgt_lang="hin")[0].cpu().numpy().squeeze()
print(tokenizer.decode(text_out, clean_up_tokenization_spaces=True, skip_special_tokens=True))

高級用法

在Fleurs數據集上推理

from datasets import load_dataset

dataset = load_dataset("google/fleurs", "hi_in", split="test")

def process_audio(example):
    audio = example["audio"]["array"]
    audio_inputs = processor(audio, sampling_rate=16_000, return_tensors="pt").to("cuda")
    text_out = model.generate(**audio_inputs, tgt_lang="hin")[0].cpu().numpy().squeeze()
    return {"predicted_text": tokenizer.decode(text_out, clean_up_tokenization_spaces=True, skip_special_tokens=True)}

dataset = dataset.map(process_audio)
dataset = dataset.remove_columns(["audio"])
dataset.to_csv("fleurs_hi_predictions.csv")

使用Fleurs進行批量翻譯

from datasets import load_dataset
import torch

def process_batch(batch):
    audio_arrays = [audio["array"] for audio in batch["audio"]]
    audio_inputs = processor(audio_arrays, sampling_rate=16_000, return_tensors="pt", padding=True).to("cuda")
    text_outs = model.generate(**audio_inputs, tgt_lang="hin")
    batch["predicted_text"] = [tokenizer.decode(text_out.cpu().numpy().squeeze(), clean_up_tokenization_spaces=True, skip_special_tokens=True) for text_out in text_outs]
    return batch

def batch_translate(language_code="hi_in", tgt_lang="hin"):
    dataset = load_dataset("google/fleurs", language_code, split="test")
    dataset = dataset.map(process_batch, batched=True, batch_size=8)
    return dataset["predicted_text"]

# Example usage
target_language = "hi_in"
translations = batch_translate(target_language, tgt_lang="hin")
print(translations)

📚 詳細文檔

本倉庫託管了IndicSeamless模型，它是基於SeamlessM4T - v2在BhasaAnuvaad數據集上進行微調的，用於跨印度語言的語音轉文本翻譯（STT）。在訓練前，數據集使用以下閾值進行了過濾：

• 對齊分數：0.8
• 挖掘分數：0.6

🔧 技術細節

該模型基於SeamlessM4T - v2進行微調，在BhasaAnuvaad數據集上進行訓練。訓練前對數據集進行了過濾，設置了對齊分數和挖掘分數的閾值，以保證數據質量。在性能上，它超越了基礎的SeamlessM4Tv2模型以及所有競爭的語音轉文本系統，在Fleurs和BhasaAnuvaad測試集上取得了優異的成績。

📄 許可證

本模型根據知識共享署名 - 非商業性使用 4.0 國際許可協議（CC BY - NC 4.0）發佈。

📄 引用

如果你在工作中使用了BhasaAnuvaad，請引用我們：

@misc{jain2024bhasaanuvaadspeechtranslationdataset,
      title={BhasaAnuvaad: A Speech Translation Dataset for 13 Indian Languages}, 
      author={Sparsh Jain and Ashwin Sankar and Devilal Choudhary and Dhairya Suman and Nikhil Narasimhan and Mohammed Safi Ur Rahman Khan and Anoop Kunchukuttan and Mitesh M Khapra and Raj Dabre},
      year={2024},
      eprint={2411.04699},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2411.04699}, 
}

📋 模型信息