Voxtral Mini開源音頻AI模型 - 免費部署實現語音轉錄、翻譯及理解

首頁

Voxtral Mini 3B 2507 Transformers

由MohamedRashad開發

Voxtral Mini是基於Ministral 3B的增強版本，具備先進的音頻輸入能力，在語音轉錄、翻譯和音頻理解等方面表現出色。

音頻生成文本

Transformers

支持多種語言開源協議:Apache-2.0 #音頻理解 #多語言轉錄 #長上下文處理

下載量 416

發布時間 : 7/18/2025

模型概述

Voxtral Mini是一個結合文本和音頻處理能力的多模態模型，保留了Ministral 3B的文本處理能力，同時增加了強大的音頻理解功能。

模型特點

專用轉錄模式

可以在純語音轉錄模式下運行，自動識別源音頻語言並進行文本轉錄

長上下文處理

支持32k標記的上下文長度，可處理長達30-40分鐘的音頻

內置問答和摘要功能

支持直接通過音頻提問並生成結構化摘要，無需單獨的ASR和語言模型

原生多語言支持

自動檢測並支持8種主要語言的音頻處理

語音直接調用功能

能夠根據語音意圖直接觸發後端功能、工作流或API調用

模型能力

語音轉錄

音頻理解

多語言支持

長音頻處理

文本生成

問答系統

摘要生成

多輪對話

使用案例

語音處理

會議記錄轉錄

將長達30分鐘的會議錄音自動轉錄為文字

高準確率的轉錄文本

多語言語音翻譯

將一種語言的語音即時翻譯為另一種語言的文本

支持8種主要語言的互譯

音頻分析

音頻內容理解

直接對音頻內容提問並獲取答案

無需先轉錄即可理解音頻內容

音頻摘要生成

分析長音頻並生成結構化摘要

節省人工整理時間

🚀 Voxtral Mini 3B - 2507（Transformers版本）

Voxtral Mini是基於Ministral 3B的增強版本，它在保留一流文本處理性能的同時，融入了先進的音頻輸入能力。該模型在語音轉錄、翻譯和音頻理解等方面表現出色。

你可以在我們的博客文章中瞭解更多關於Voxtral的信息。

✨ 主要特性

Voxtral在Ministral - 3B的基礎上，具備強大的音頻理解能力：

專用轉錄模式：Voxtral可以在純語音轉錄模式下運行，以實現最佳性能。默認情況下，Voxtral會自動識別源音頻語言，並相應地進行文本轉錄。
長上下文處理：擁有32k的標記上下文長度，Voxtral能夠處理長達30分鐘的音頻進行轉錄，或40分鐘的音頻進行理解。
內置問答和摘要功能：支持直接通過音頻提問。可以分析音頻並生成結構化摘要，無需單獨的自動語音識別（ASR）和語言模型。
原生多語言支持：自動檢測語言，並在世界上使用最廣泛的語言（英語、西班牙語、法語、葡萄牙語、印地語、德語、荷蘭語、意大利語）中表現出色。
語音直接調用功能：能夠根據用戶的語音意圖直接觸發後端功能、工作流或API調用。
強大的文本處理能力：保留了其語言模型主幹Ministral - 3B的文本理解能力。

📊 基準測試結果

音頻

在FLEURS、Mozilla Common Voice和Multilingual LibriSpeech基準測試中的平均單詞錯誤率（WER）：

image/png

文本

image/png

📦 安裝指南

安裝Transformers（從源代碼安裝）

pip install git+https://github.com/huggingface/transformers

💻 使用示例

該模型可以與以下框架一起使用：

Transformers 🤗：詳見此處

⚠️ 重要提示

在聊天完成任務（例如音頻理解）中，建議使用temperature = 0.2和top_p = 0.95；在轉錄任務中，建議使用temperature = 0.0。

支持每條消息包含多個音頻，以及多輪帶音頻的用戶對話。

目前暫不支持系統提示。

Transformers 🤗

Voxtral原生支持Transformers！

音頻指令

➡️ 多音頻 + 文本指令

from transformers import VoxtralForConditionalGeneration, AutoProcessor
import torch

device = "cuda"
repo_id = "MohamedRashad/Voxtral-Mini-3B-2507-transformers"

processor = AutoProcessor.from_pretrained(repo_id)
model = VoxtralForConditionalGeneration.from_pretrained(repo_id, torch_dtype=torch.bfloat16, device_map=device)

conversation = [
    {
        "role": "user",
        "content": [
            {
                "type": "audio",
                "path": "https://huggingface.co/datasets/hf-internal-testing/dummy-audio-samples/resolve/main/mary_had_lamb.mp3",
            },
            {
                "type": "audio",
                "path": "https://huggingface.co/datasets/hf-internal-testing/dummy-audio-samples/resolve/main/winning_call.mp3",
            },
            {"type": "text", "text": "What sport and what nursery rhyme are referenced?"},
        ],
    }
]

inputs = processor.apply_chat_template(conversation)
inputs = inputs.to(device, dtype=torch.bfloat16)

outputs = model.generate(**inputs, max_new_tokens=500)
decoded_outputs = processor.batch_decode(outputs[:, inputs.input_ids.shape[1]:], skip_special_tokens=True)

print("\nGenerated response:")
print("=" * 80)
print(decoded_outputs[0])
print("=" * 80)

➡️ 多輪對話

from transformers import VoxtralForConditionalGeneration, AutoProcessor
import torch

device = "cuda"
repo_id = "MohamedRashad/Voxtral-Mini-3B-2507-transformers"

processor = AutoProcessor.from_pretrained(repo_id)
model = VoxtralForConditionalGeneration.from_pretrained(repo_id, torch_dtype=torch.bfloat16, device_map=device)

conversation = [
    {
        "role": "user",
        "content": [
            {
                "type": "audio",
                "path": "https://huggingface.co/datasets/hf-internal-testing/dummy-audio-samples/resolve/main/obama.mp3",
            },
            {
                "type": "audio",
                "path": "https://huggingface.co/datasets/hf-internal-testing/dummy-audio-samples/resolve/main/bcn_weather.mp3",
            },
            {"type": "text", "text": "Describe briefly what you can hear."},
        ],
    },
    {
        "role": "assistant",
        "content": "The audio begins with the speaker delivering a farewell address in Chicago, reflecting on his eight years as president and expressing gratitude to the American people. The audio then transitions to a weather report, stating that it was 35 degrees in Barcelona the previous day, but the temperature would drop to minus 20 degrees the following day.",
    },
    {
        "role": "user",
        "content": [
            {
                "type": "audio",
                "path": "https://huggingface.co/datasets/hf-internal-testing/dummy-audio-samples/resolve/main/winning_call.mp3",
            },
            {"type": "text", "text": "Ok, now compare this new audio with the previous one."},
        ],
    },
]

inputs = processor.apply_chat_template(conversation)
inputs = inputs.to(device, dtype=torch.bfloat16)

outputs = model.generate(**inputs, max_new_tokens=500)
decoded_outputs = processor.batch_decode(outputs[:, inputs.input_ids.shape[1]:], skip_special_tokens=True)

print("\nGenerated response:")
print("=" * 80)
print(decoded_outputs[0])
print("=" * 80)

➡️ 僅文本輸入

from transformers import VoxtralForConditionalGeneration, AutoProcessor
import torch

device = "cuda"
repo_id = "MohamedRashad/Voxtral-Mini-3B-2507-transformers"

processor = AutoProcessor.from_pretrained(repo_id)
model = VoxtralForConditionalGeneration.from_pretrained(repo_id, torch_dtype=torch.bfloat16, device_map=device)

conversation = [
    {
        "role": "user",
        "content": [
            {
                "type": "text",
                "text": "Why should AI models be open-sourced?",
            },
        ],
    }
]

inputs = processor.apply_chat_template(conversation)
inputs = inputs.to(device, dtype=torch.bfloat16)

outputs = model.generate(**inputs, max_new_tokens=500)
decoded_outputs = processor.batch_decode(outputs[:, inputs.input_ids.shape[1]:], skip_special_tokens=True)

print("\nGenerated response:")
print("=" * 80)
print(decoded_outputs[0])
print("=" * 80)

➡️ 僅音頻輸入

from transformers import VoxtralForConditionalGeneration, AutoProcessor
import torch

device = "cuda"
repo_id = "MohamedRashad/Voxtral-Mini-3B-2507-transformers"

processor = AutoProcessor.from_pretrained(repo_id)
model = VoxtralForConditionalGeneration.from_pretrained(repo_id, torch_dtype=torch.bfloat16, device_map=device)

conversation = [
    {
        "role": "user",
        "content": [
            {
                "type": "audio",
                "path": "https://huggingface.co/datasets/hf-internal-testing/dummy-audio-samples/resolve/main/winning_call.mp3",
            },
        ],
    }
]

inputs = processor.apply_chat_template(conversation)
inputs = inputs.to(device, dtype=torch.bfloat16)

outputs = model.generate(**inputs, max_new_tokens=500)
decoded_outputs = processor.batch_decode(outputs[:, inputs.input_ids.shape[1]:], skip_special_tokens=True)

print("\nGenerated response:")
print("=" * 80)
print(decoded_outputs[0])
print("=" * 80)

➡️ 批量推理

from transformers import VoxtralForConditionalGeneration, AutoProcessor
import torch

device = "cuda"
repo_id = "MohamedRashad/Voxtral-Mini-3B-2507-transformers"

processor = AutoProcessor.from_pretrained(repo_id)
model = VoxtralForConditionalGeneration.from_pretrained(repo_id, torch_dtype=torch.bfloat16, device_map=device)

conversations = [
    [
        {
            "role": "user",
            "content": [
                {
                    "type": "audio",
                    "path": "https://huggingface.co/datasets/hf-internal-testing/dummy-audio-samples/resolve/main/obama.mp3",
                },
                {
                    "type": "audio",
                    "path": "https://huggingface.co/datasets/hf-internal-testing/dummy-audio-samples/resolve/main/bcn_weather.mp3",
                },
                {
                    "type": "text",
                    "text": "Who's speaking in the speach and what city's weather is being discussed?",
                },
            ],
        }
    ],
    [
        {
            "role": "user",
            "content": [
                {
                    "type": "audio",
                    "path": "https://huggingface.co/datasets/hf-internal-testing/dummy-audio-samples/resolve/main/winning_call.mp3",
                },
                {"type": "text", "text": "What can you tell me about this audio?"},
            ],
        }
    ],
]

inputs = processor.apply_chat_template(conversations)
inputs = inputs.to(device, dtype=torch.bfloat16)

outputs = model.generate(**inputs, max_new_tokens=500)
decoded_outputs = processor.batch_decode(outputs[:, inputs.input_ids.shape[1]:], skip_special_tokens=True)

print("\nGenerated responses:")
print("=" * 80)
for decoded_output in decoded_outputs:
    print(decoded_output)
    print("=" * 80)

轉錄

➡️ 轉錄示例

from transformers import VoxtralForConditionalGeneration, AutoProcessor
import torch

device = "cuda"
repo_id = "MohamedRashad/Voxtral-Mini-3B-2507-transformers"

processor = AutoProcessor.from_pretrained(repo_id)
model = VoxtralForConditionalGeneration.from_pretrained(repo_id, torch_dtype=torch.bfloat16, device_map=device)

inputs = processor.apply_transcrition_request(language="en", audio="https://huggingface.co/datasets/hf-internal-testing/dummy-audio-samples/resolve/main/obama.mp3", model_id=repo_id)
inputs = inputs.to(device, dtype=torch.bfloat16)

outputs = model.generate(**inputs, max_new_tokens=500)
decoded_outputs = processor.batch_decode(outputs[:, inputs.input_ids.shape[1]:], skip_special_tokens=True)

print("\nGenerated responses:")
print("=" * 80)
for decoded_output in decoded_outputs:
    print(decoded_output)
    print("=" * 80)