Granite Speech 3.3 8b

🚀 Granite-speech-3.3-8b

Granite-speech-3.3-8b是一款紧凑高效的语音语言模型，专为自动语音识别（ASR）和自动语音翻译（AST）而设计。它采用双通设计，与将语音和语言处理整合为单通的集成模型不同。首次调用该模型时，它会将音频文件转录为文本；若要使用底层的Granite语言模型处理转录后的文本，用户需进行第二次调用，因为每个步骤都必须显式启动。

🚀 快速开始

Granite Speech模型在transformers库的main分支中得到原生支持。以下是使用granite-speech-3.3-8b模型的简单示例。

使用transformers库

首先，确保从源代码构建最新版本的transformers库：

pip install https://github.com/huggingface/transformers/archive/main.zip torchaudio peft soundfile

然后运行以下代码：

import torch
import torchaudio
from transformers import AutoProcessor, AutoModelForSpeechSeq2Seq
from huggingface_hub import hf_hub_download

device = "cuda" if torch.cuda.is_available() else "cpu"

model_name = "ibm-granite/granite-speech-3.3-8b"
speech_granite_processor = AutoProcessor.from_pretrained(
    model_name)
tokenizer = speech_granite_processor.tokenizer
speech_granite = AutoModelForSpeechSeq2Seq.from_pretrained(
    model_name).to(device)

# prepare speech and text prompt, using the appropriate prompt template

audio_path = hf_hub_download(repo_id=model_name, filename='10226_10111_000000.wav')
wav, sr = torchaudio.load(audio_path, normalize=True)
assert wav.shape[0] == 1 and sr == 16000 # mono, 16khz

# create text prompt
chat = [
    {
        "role": "system",
        "content": "Knowledge Cutoff Date: April 2024.\nToday's Date: April 9, 2025.\nYou are Granite, developed by IBM. You are a helpful AI assistant",
    },
    {
        "role": "user",
        "content": "<|audio|>can you transcribe the speech into a written format?",
    }
]

text = tokenizer.apply_chat_template(
    chat, tokenize=False, add_generation_prompt=True
)

# compute audio embeddings
model_inputs = speech_granite_processor(
    text,
    wav,
    device=device, # Computation device; returned tensors are put on CPU
    return_tensors="pt",
).to(device)
 
model_outputs = speech_granite.generate(
    **model_inputs,
    max_new_tokens=200,
    num_beams=4,
    do_sample=False,
    min_length=1,
    top_p=1.0,
    repetition_penalty=1.0,
    length_penalty=1.0,
    temperature=1.0,
    bos_token_id=tokenizer.bos_token_id,
    eos_token_id=tokenizer.eos_token_id,
    pad_token_id=tokenizer.pad_token_id,
)

# Transformers includes the input IDs in the response.
num_input_tokens = model_inputs["input_ids"].shape[-1]
new_tokens = torch.unsqueeze(model_outputs[0, num_input_tokens:], dim=0)

output_text = tokenizer.batch_decode(
    new_tokens, add_special_tokens=False, skip_special_tokens=True
)
print(f"STT output = {output_text[0].upper()}")

使用vLLM库

首先，确保安装最新版本的vLLM库：

pip install vllm --upgrade

离线模式代码

from transformers import AutoTokenizer
from vllm import LLM, SamplingParams
from vllm.assets.audio import AudioAsset
from vllm.lora.request import LoRARequest

model_id = "ibm-granite/granite-speech-3.3-8b"
tokenizer = AutoTokenizer.from_pretrained(model_id)

def get_prompt(question: str, has_audio: bool):
    """Build the input prompt to send to vLLM."""
    if has_audio:
        question = f"<|audio|>{question}"
    chat = [
        {
            "role": "user",
            "content": question
        }
    ]
    return tokenizer.apply_chat_template(chat, tokenize=False)

# NOTE - you may see warnings about multimodal lora layers being ignored;
# this is okay as the lora in this model is only applied to the LLM.
model = LLM(
    model=model_id,
    enable_lora=True,
    max_lora_rank=64,
    max_model_len=2048, # This may be needed for lower resource devices.
    limit_mm_per_prompt={"audio": 1},
)

### 1. Example with Audio [make sure to use the lora]
question = "can you transcribe the speech into a written format?"
prompt_with_audio = get_prompt(
    question=question,
    has_audio=True,
)
audio = AudioAsset("mary_had_lamb").audio_and_sample_rate

inputs = {
    "prompt": prompt_with_audio,
    "multi_modal_data": {
        "audio": audio,
    }
}

outputs = model.generate(
    inputs,
    sampling_params=SamplingParams(
        temperature=0.2,
        max_tokens=64,
    ),
    lora_request=[LoRARequest("speech", 1, model_id)]
)
print(f"Audio Example - Question: {question}")
print(f"Generated text: {outputs[0].outputs[0].text}")


### 2. Example without Audio [do NOT use the lora]
question = "What is the capital of Brazil?"
prompt = get_prompt(
    question=question,
    has_audio=False,
)

outputs = model.generate(
    {"prompt": prompt},
    sampling_params=SamplingParams(
        temperature=0.2,
        max_tokens=12,
    ),
)
print(f"Text Only Example - Question: {question}")
print(f"Generated text: {outputs[0].outputs[0].text}")

在线模式代码

"""
Launch the vLLM server with the following command:

vllm serve ibm-granite/granite-speech-3.3-8b \
    --api-key token-abc123 \
    --max-model-len 2048 \
    --enable-lora  \
    --lora-modules speech=ibm-granite/granite-speech-3.3-8b \
    --max-lora-rank 64
"""

import base64

import requests
from openai import OpenAI

from vllm.assets.audio import AudioAsset

# Modify OpenAI's API key and API base to use vLLM's API server.
openai_api_key = "token-abc123"
openai_api_base = "http://localhost:8000/v1"

client = OpenAI(
    # defaults to os.environ.get("OPENAI_API_KEY")
    api_key=openai_api_key,
    base_url=openai_api_base,
)

base_model_name = "ibm-granite/granite-speech-3.3-8b"
lora_model_name = "speech"
# Any format supported by librosa is supported
audio_url = AudioAsset("mary_had_lamb").url

# Use base64 encoded audio in the payload
def encode_audio_base64_from_url(audio_url: str) -> str:
    """Encode an audio retrieved from a remote url to base64 format."""
    with requests.get(audio_url) as response:
        response.raise_for_status()
        result = base64.b64encode(response.content).decode('utf-8')
    return result

audio_base64 = encode_audio_base64_from_url(audio_url=audio_url)

### 1. Example with Audio
# NOTE: we pass the name of the lora model (`speech`) here because we have audio.
question = "can you transcribe the speech into a written format?"
chat_completion_with_audio = client.chat.completions.create(
    messages=[{
        "role": "user",
        "content": [
            {
                "type": "text",
                "text": question
            },
            {
                "type": "audio_url",
                "audio_url": {
                    # Any format supported by librosa is supported
                    "url": f"data:audio/ogg;base64,{audio_base64}"
                },
            },
        ],
    }],
    temperature=0.2,
    max_tokens=64,
    model=lora_model_name,
)


print(f"Audio Example - Question: {question}")
print(f"Generated text: {chat_completion_with_audio.choices[0].message.content}")


### 2. Example without Audio
# NOTE: we pass the name of the base model here because we do not have audio.
question = "What is the capital of Brazil?"
chat_completion_with_audio = client.chat.completions.create(
    messages=[{
        "role": "user",
        "content": [
            {
                "type": "text",
                "text": question
            },
        ],
    }],
    temperature=0.2,
    max_tokens=12,
    model=base_model_name,
)

print(f"Text Only Example - Question: {question}")
print(f"Generated text: {chat_completion_with_audio.choices[0].message.content}")

✨ 主要特性

• 紧凑高效：专为自动语音识别（ASR）和自动语音翻译（AST）设计，采用双通设计，提高处理效率。
• 多模态适配：通过语音编码器、语音投影器和时间下采样器等组件，实现语音和文本模态的有效适配。
• 可扩展性：基于IBM的超级计算集群进行训练，具备可扩展的训练基础设施。

📦 安装指南

使用transformers库时，需从源代码构建最新版本：

pip install https://github.com/huggingface/transformers/archive/main.zip torchaudio peft soundfile

使用vLLM库时，需安装最新版本：

pip install vllm --upgrade

📚 详细文档

模型架构

Granite-speech-3.3-8b的架构由以下组件组成：

1. 语音编码器：包含10个Conformer块，使用连接主义时序分类（CTC）在字符级目标上进行训练。 | 配置参数 | 值 | | ---- | ---- | | 输入维度 | 160 (80 logmels x 2) | | 层数 | 10 | | 隐藏维度 | 1024 | | 注意力头数量 | 8 | | 注意力头大小 | 128 | | 卷积核大小 | 15 | | 输出维度 | 42 |

2. 语音投影器和时间下采样器（语音 - 文本模态适配器）：使用2层窗口查询变压器（q-former），对来自语音编码器最后一个Conformer块的15个1024维声学嵌入块进行操作，通过每层每个块使用3个可训练查询将其下采样5倍。

3. 大语言模型：采用具有128k上下文长度的granite-3.3-8b-instruct。

4. LoRA适配器：秩为64，应用于查询、值投影矩阵。

训练数据

训练数据主要来自两个关键来源：公开可用数据集和针对语音翻译任务创建的合成数据。

基础设施

使用IBM的超级计算集群Blue Vela进行训练，该集群配备NVIDIA H100 GPU，可在数千个GPU上提供可扩展且高效的训练基础设施。此特定模型在32个H100 GPU上训练9天完成。

🔧 技术细节

评估

在标准基准测试中，将Granite-speech-3.3-8b与参数少于80亿的其他语音语言模型（SLM）以及专用的ASR和AST系统进行了评估。评估涵盖多个公共基准，特别侧重于英语ASR任务，同时也包括英语到其他语言（En-X）的翻译任务。

📄 许可证

本模型采用Apache 2.0许可证。

⚠️ 重要提示

• 模型在使用num_beams=1进行贪心解码或处理极短音频片段（<0.1s）时可能产生不可靠输出。在进一步更新发布之前，建议使用大于1的束搜索大小，并避免输入低于0.1秒的音频，以确保更一致的性能。
• 大语音和语言模型的使用可能涉及风险和伦理考量，包括偏差和公平性、错误信息以及自主决策等问题。建议社区按照IBM的负责任使用指南或类似的负责任使用框架使用Granite-speech-3.3-8b。

💡 使用建议

为增强安全性，建议将Granite-speech-3.3-8b与Granite Guardian一起使用。Granite Guardian是一个经过微调的指令模型，旨在检测和标记提示和响应中的风险。

📦 资源

• 📄 阅读完整技术报告：https://arxiv.org/abs/2505.08699
• ⭐️ 了解Granite的最新更新：https://www.ibm.com/granite
• 🚀 获取教程、最佳实践和提示工程建议：https://www.ibm.com/granite/docs/
• 💡 了解最新的Granite学习资源：https://ibm.biz/granite-learning-resources