模型简介
模型特点
模型能力
使用案例
🚀 Granite-speech-3.3-8b
Granite-speech-3.3-8b是一款紧凑高效的语音语言模型,专为自动语音识别(ASR)和自动语音翻译(AST)而设计。它采用双通设计,与将语音和语言处理整合为单通的集成模型不同。首次调用该模型时,它会将音频文件转录为文本;若要使用底层的Granite语言模型处理转录后的文本,用户需进行第二次调用,因为每个步骤都必须显式启动。
🚀 快速开始
Granite Speech模型在transformers
库的main
分支中得到原生支持。以下是使用granite-speech-3.3-8b
模型的简单示例。
使用transformers
库
首先,确保从源代码构建最新版本的transformers
库:
pip install https://github.com/huggingface/transformers/archive/main.zip torchaudio peft soundfile
然后运行以下代码:
import torch
import torchaudio
from transformers import AutoProcessor, AutoModelForSpeechSeq2Seq
from huggingface_hub import hf_hub_download
device = "cuda" if torch.cuda.is_available() else "cpu"
model_name = "ibm-granite/granite-speech-3.3-8b"
speech_granite_processor = AutoProcessor.from_pretrained(
model_name)
tokenizer = speech_granite_processor.tokenizer
speech_granite = AutoModelForSpeechSeq2Seq.from_pretrained(
model_name).to(device)
# prepare speech and text prompt, using the appropriate prompt template
audio_path = hf_hub_download(repo_id=model_name, filename='10226_10111_000000.wav')
wav, sr = torchaudio.load(audio_path, normalize=True)
assert wav.shape[0] == 1 and sr == 16000 # mono, 16khz
# create text prompt
chat = [
{
"role": "system",
"content": "Knowledge Cutoff Date: April 2024.\nToday's Date: April 9, 2025.\nYou are Granite, developed by IBM. You are a helpful AI assistant",
},
{
"role": "user",
"content": "<|audio|>can you transcribe the speech into a written format?",
}
]
text = tokenizer.apply_chat_template(
chat, tokenize=False, add_generation_prompt=True
)
# compute audio embeddings
model_inputs = speech_granite_processor(
text,
wav,
device=device, # Computation device; returned tensors are put on CPU
return_tensors="pt",
).to(device)
model_outputs = speech_granite.generate(
**model_inputs,
max_new_tokens=200,
num_beams=4,
do_sample=False,
min_length=1,
top_p=1.0,
repetition_penalty=1.0,
length_penalty=1.0,
temperature=1.0,
bos_token_id=tokenizer.bos_token_id,
eos_token_id=tokenizer.eos_token_id,
pad_token_id=tokenizer.pad_token_id,
)
# Transformers includes the input IDs in the response.
num_input_tokens = model_inputs["input_ids"].shape[-1]
new_tokens = torch.unsqueeze(model_outputs[0, num_input_tokens:], dim=0)
output_text = tokenizer.batch_decode(
new_tokens, add_special_tokens=False, skip_special_tokens=True
)
print(f"STT output = {output_text[0].upper()}")
使用vLLM
库
首先,确保安装最新版本的vLLM
库:
pip install vllm --upgrade
离线模式代码
from transformers import AutoTokenizer
from vllm import LLM, SamplingParams
from vllm.assets.audio import AudioAsset
from vllm.lora.request import LoRARequest
model_id = "ibm-granite/granite-speech-3.3-8b"
tokenizer = AutoTokenizer.from_pretrained(model_id)
def get_prompt(question: str, has_audio: bool):
"""Build the input prompt to send to vLLM."""
if has_audio:
question = f"<|audio|>{question}"
chat = [
{
"role": "user",
"content": question
}
]
return tokenizer.apply_chat_template(chat, tokenize=False)
# NOTE - you may see warnings about multimodal lora layers being ignored;
# this is okay as the lora in this model is only applied to the LLM.
model = LLM(
model=model_id,
enable_lora=True,
max_lora_rank=64,
max_model_len=2048, # This may be needed for lower resource devices.
limit_mm_per_prompt={"audio": 1},
)
### 1. Example with Audio [make sure to use the lora]
question = "can you transcribe the speech into a written format?"
prompt_with_audio = get_prompt(
question=question,
has_audio=True,
)
audio = AudioAsset("mary_had_lamb").audio_and_sample_rate
inputs = {
"prompt": prompt_with_audio,
"multi_modal_data": {
"audio": audio,
}
}
outputs = model.generate(
inputs,
sampling_params=SamplingParams(
temperature=0.2,
max_tokens=64,
),
lora_request=[LoRARequest("speech", 1, model_id)]
)
print(f"Audio Example - Question: {question}")
print(f"Generated text: {outputs[0].outputs[0].text}")
### 2. Example without Audio [do NOT use the lora]
question = "What is the capital of Brazil?"
prompt = get_prompt(
question=question,
has_audio=False,
)
outputs = model.generate(
{"prompt": prompt},
sampling_params=SamplingParams(
temperature=0.2,
max_tokens=12,
),
)
print(f"Text Only Example - Question: {question}")
print(f"Generated text: {outputs[0].outputs[0].text}")
在线模式代码
"""
Launch the vLLM server with the following command:
vllm serve ibm-granite/granite-speech-3.3-8b \
--api-key token-abc123 \
--max-model-len 2048 \
--enable-lora \
--lora-modules speech=ibm-granite/granite-speech-3.3-8b \
--max-lora-rank 64
"""
import base64
import requests
from openai import OpenAI
from vllm.assets.audio import AudioAsset
# Modify OpenAI's API key and API base to use vLLM's API server.
openai_api_key = "token-abc123"
openai_api_base = "http://localhost:8000/v1"
client = OpenAI(
# defaults to os.environ.get("OPENAI_API_KEY")
api_key=openai_api_key,
base_url=openai_api_base,
)
base_model_name = "ibm-granite/granite-speech-3.3-8b"
lora_model_name = "speech"
# Any format supported by librosa is supported
audio_url = AudioAsset("mary_had_lamb").url
# Use base64 encoded audio in the payload
def encode_audio_base64_from_url(audio_url: str) -> str:
"""Encode an audio retrieved from a remote url to base64 format."""
with requests.get(audio_url) as response:
response.raise_for_status()
result = base64.b64encode(response.content).decode('utf-8')
return result
audio_base64 = encode_audio_base64_from_url(audio_url=audio_url)
### 1. Example with Audio
# NOTE: we pass the name of the lora model (`speech`) here because we have audio.
question = "can you transcribe the speech into a written format?"
chat_completion_with_audio = client.chat.completions.create(
messages=[{
"role": "user",
"content": [
{
"type": "text",
"text": question
},
{
"type": "audio_url",
"audio_url": {
# Any format supported by librosa is supported
"url": f"data:audio/ogg;base64,{audio_base64}"
},
},
],
}],
temperature=0.2,
max_tokens=64,
model=lora_model_name,
)
print(f"Audio Example - Question: {question}")
print(f"Generated text: {chat_completion_with_audio.choices[0].message.content}")
### 2. Example without Audio
# NOTE: we pass the name of the base model here because we do not have audio.
question = "What is the capital of Brazil?"
chat_completion_with_audio = client.chat.completions.create(
messages=[{
"role": "user",
"content": [
{
"type": "text",
"text": question
},
],
}],
temperature=0.2,
max_tokens=12,
model=base_model_name,
)
print(f"Text Only Example - Question: {question}")
print(f"Generated text: {chat_completion_with_audio.choices[0].message.content}")
✨ 主要特性
- 紧凑高效:专为自动语音识别(ASR)和自动语音翻译(AST)设计,采用双通设计,提高处理效率。
- 多模态适配:通过语音编码器、语音投影器和时间下采样器等组件,实现语音和文本模态的有效适配。
- 可扩展性:基于IBM的超级计算集群进行训练,具备可扩展的训练基础设施。
📦 安装指南
使用transformers
库时,需从源代码构建最新版本:
pip install https://github.com/huggingface/transformers/archive/main.zip torchaudio peft soundfile
使用vLLM
库时,需安装最新版本:
pip install vllm --upgrade
📚 详细文档
模型架构
Granite-speech-3.3-8b的架构由以下组件组成:
-
语音编码器:包含10个Conformer块,使用连接主义时序分类(CTC)在字符级目标上进行训练。 | 配置参数 | 值 | | ---- | ---- | | 输入维度 | 160 (80 logmels x 2) | | 层数 | 10 | | 隐藏维度 | 1024 | | 注意力头数量 | 8 | | 注意力头大小 | 128 | | 卷积核大小 | 15 | | 输出维度 | 42 |
-
语音投影器和时间下采样器(语音 - 文本模态适配器):使用2层窗口查询变压器(q-former),对来自语音编码器最后一个Conformer块的15个1024维声学嵌入块进行操作,通过每层每个块使用3个可训练查询将其下采样5倍。
-
大语言模型:采用具有128k上下文长度的granite-3.3-8b-instruct。
-
LoRA适配器:秩为64,应用于查询、值投影矩阵。
训练数据
训练数据主要来自两个关键来源:公开可用数据集和针对语音翻译任务创建的合成数据。
名称 | 任务 | 时长(小时) | 来源 |
---|---|---|---|
CommonVoice-17 English | ASR | 2600 | https://huggingface.co/datasets/mozilla-foundation/common_voice_17_0 |
MLS English | ASR | 44000 | https://huggingface.co/datasets/facebook/multilingual_librispeech |
Librispeech | ASR | 1000 | https://huggingface.co/datasets/openslr/librispeech_asr |
VoxPopuli English | ASR | 500 | https://huggingface.co/datasets/facebook/voxpopuli |
AMI | ASR | 100 | https://huggingface.co/datasets/edinburghcstr/ami |
YODAS English | ASR | 10000 | https://huggingface.co/datasets/espnet/yodas |
Switchboard English | ASR | 260 | https://catalog.ldc.upenn.edu/LDC97S62 |
CallHome English | ASR | 18 | https://catalog.ldc.upenn.edu/LDC97T14 |
Fisher | ASR | 2000 | https://catalog.ldc.upenn.edu/LDC2004S13 |
Voicemail part I | ASR | 40 | https://catalog.ldc.upenn.edu/LDC98S77 |
Voicemail part II | ASR | 40 | https://catalog.ldc.upenn.edu/LDC2002S35 |
CommonVoice-17 En->De,Es,Fr,It,Ja,Pt,Zh | AST | 2600*7 | Translations with Phi-4 and MADLAD |
基础设施
使用IBM的超级计算集群Blue Vela进行训练,该集群配备NVIDIA H100 GPU,可在数千个GPU上提供可扩展且高效的训练基础设施。此特定模型在32个H100 GPU上训练9天完成。
🔧 技术细节
评估
在标准基准测试中,将Granite-speech-3.3-8b与参数少于80亿的其他语音语言模型(SLM)以及专用的ASR和AST系统进行了评估。评估涵盖多个公共基准,特别侧重于英语ASR任务,同时也包括英语到其他语言(En-X)的翻译任务。
📄 许可证
本模型采用Apache 2.0许可证。
⚠️ 重要提示
- 模型在使用
num_beams=1
进行贪心解码或处理极短音频片段(<0.1s)时可能产生不可靠输出。在进一步更新发布之前,建议使用大于1的束搜索大小,并避免输入低于0.1秒的音频,以确保更一致的性能。 - 大语音和语言模型的使用可能涉及风险和伦理考量,包括偏差和公平性、错误信息以及自主决策等问题。建议社区按照IBM的负责任使用指南或类似的负责任使用框架使用Granite-speech-3.3-8b。
💡 使用建议
为增强安全性,建议将Granite-speech-3.3-8b与Granite Guardian一起使用。Granite Guardian是一个经过微调的指令模型,旨在检测和标记提示和响应中的风险。
📦 资源
- 📄 阅读完整技术报告:https://arxiv.org/abs/2505.08699
- ⭐️ 了解Granite的最新更新:https://www.ibm.com/granite
- 🚀 获取教程、最佳实践和提示工程建议:https://www.ibm.com/granite/docs/
- 💡 了解最新的Granite学习资源:https://ibm.biz/granite-learning-resources











