Granite Speech 3.3 2b

🚀 Granite-speech-3.3-2b

Granite-speech-3.3-2b是一款紧凑高效的语音语言模型，专为自动语音识别（ASR）和自动语音翻译（AST）而设计。该模型采用双通设计，与将语音和语言处理整合为单通的集成模型不同。首次调用时，它会将音频文件转录为文本；若要使用底层的Granite语言模型处理转录后的文本，用户需进行第二次调用，因为每个步骤都必须明确启动。

🚀 快速开始

Granite语音模型在transformers库的main分支中得到原生支持。以下是使用granite-speech-3.3-2b模型的简单示例。

使用transformers库

首先，确保从源代码构建最新版本的transformers库：

pip install https://github.com/huggingface/transformers/archive/main.zip torchaudio peft soundfile

然后运行以下代码：

import torch
import torchaudio
from transformers import AutoProcessor, AutoModelForSpeechSeq2Seq
from huggingface_hub import hf_hub_download

device = "cuda" if torch.cuda.is_available() else "cpu"

model_name = "ibm-granite/granite-speech-3.3-2b"
speech_granite_processor = AutoProcessor.from_pretrained(
    model_name)
tokenizer = speech_granite_processor.tokenizer
speech_granite = AutoModelForSpeechSeq2Seq.from_pretrained(
    model_name).to(device)

# prepare speech and text prompt, using the appropriate prompt template

audio_path = hf_hub_download(repo_id=model_name, filename='10226_10111_000000.wav')
wav, sr = torchaudio.load(audio_path, normalize=True)
assert wav.shape[0] == 1 and sr == 16000 # mono, 16khz

# create text prompt
chat = [
    {
        "role": "system",
        "content": "Knowledge Cutoff Date: April 2024.\nToday's Date: May 2, 2025.\nYou are Granite, developed by IBM. You are a helpful AI assistant",
    },
    {
        "role": "user",
        "content": "<|audio|>can you transcribe the speech into a written format?",
    }
]

text = tokenizer.apply_chat_template(
    chat, tokenize=False, add_generation_prompt=True
)

# compute audio embeddings
model_inputs = speech_granite_processor(
    text,
    wav,
    device=device, # Computation device; returned tensors are put on CPU
    return_tensors="pt",
).to(device)
 
model_outputs = speech_granite.generate(
    **model_inputs,
    max_new_tokens=200,
    num_beams=4,
    do_sample=False,
    min_length=1,
    top_p=1.0,
    repetition_penalty=1.0,
    length_penalty=1.0,
    temperature=1.0,
    bos_token_id=tokenizer.bos_token_id,
    eos_token_id=tokenizer.eos_token_id,
    pad_token_id=tokenizer.pad_token_id,
)

# Transformers includes the input IDs in the response.
num_input_tokens = model_inputs["input_ids"].shape[-1]
new_tokens = torch.unsqueeze(model_outputs[0, num_input_tokens:], dim=0)

output_text = tokenizer.batch_decode(
    new_tokens, add_special_tokens=False, skip_special_tokens=True
)
print(f"STT output = {output_text[0].upper()}")

使用vLLM库

首先，确保安装最新版本的vLLM库：

pip install vllm --upgrade

离线模式代码

from transformers import AutoTokenizer
from vllm import LLM, SamplingParams
from vllm.assets.audio import AudioAsset
from vllm.lora.request import LoRARequest

model_id = "ibm-granite/granite-speech-3.3-2b"
tokenizer = AutoTokenizer.from_pretrained(model_id)

def get_prompt(question: str, has_audio: bool):
    """Build the input prompt to send to vLLM."""
    if has_audio:
        question = f"<|audio|>{question}"
    chat = [
        {
            "role": "user",
            "content": question
        }
    ]
    return tokenizer.apply_chat_template(chat, tokenize=False)

# NOTE - you may see warnings about multimodal lora layers being ignored;
# this is okay as the lora in this model is only applied to the LLM.
model = LLM(
    model=model_id,
    enable_lora=True,
    max_lora_rank=64,
    max_model_len=2048, # This may be needed for lower resource devices.
    limit_mm_per_prompt={"audio": 1},
)

### 1. Example with Audio [make sure to use the lora]
question = "can you transcribe the speech into a written format?"
prompt_with_audio = get_prompt(
    question=question,
    has_audio=True,
)
audio = AudioAsset("mary_had_lamb").audio_and_sample_rate

inputs = {
    "prompt": prompt_with_audio,
    "multi_modal_data": {
        "audio": audio,
    }
}

outputs = model.generate(
    inputs,
    sampling_params=SamplingParams(
        temperature=0.2,
        max_tokens=64,
    ),
    lora_request=[LoRARequest("speech", 1, model_id)]
)
print(f"Audio Example - Question: {question}")
print(f"Generated text: {outputs[0].outputs[0].text}")


### 2. Example without Audio [do NOT use the lora]
question = "What is the capital of Brazil?"
prompt = get_prompt(
    question=question,
    has_audio=False,
)

outputs = model.generate(
    {"prompt": prompt},
    sampling_params=SamplingParams(
        temperature=0.2,
        max_tokens=12,
    ),
)
print(f"Text Only Example - Question: {question}")
print(f"Generated text: {outputs[0].outputs[0].text}")

在线模式代码

"""
Launch the vLLM server with the following command:

vllm serve ibm-granite/granite-speech-3.3-2b \
    --api-key token-abc123 \
    --max-model-len 2048 \
    --enable-lora  \
    --lora-modules speech=ibm-granite/granite-speech-3.3-2b \
    --max-lora-rank 64
"""

import base64

import requests
from openai import OpenAI

from vllm.assets.audio import AudioAsset

# Modify OpenAI's API key and API base to use vLLM's API server.
openai_api_key = "token-abc123"
openai_api_base = "http://localhost:8000/v1"

client = OpenAI(
    # defaults to os.environ.get("OPENAI_API_KEY")
    api_key=openai_api_key,
    base_url=openai_api_base,
)

base_model_name = "ibm-granite/granite-speech-3.3-2b"
lora_model_name = "speech"
# Any format supported by librosa is supported
audio_url = AudioAsset("mary_had_lamb").url

# Use base64 encoded audio in the payload
def encode_audio_base64_from_url(audio_url: str) -> str:
    """Encode an audio retrieved from a remote url to base64 format."""
    with requests.get(audio_url) as response:
        response.raise_for_status()
        result = base64.b64encode(response.content).decode('utf-8')
    return result

audio_base64 = encode_audio_base64_from_url(audio_url=audio_url)

### 1. Example with Audio
# NOTE: we pass the name of the lora model (`speech`) here because we have audio.
question = "can you transcribe the speech into a written format?"
chat_completion_with_audio = client.chat.completions.create(
    messages=[{
        "role": "user",
        "content": [
            {
                "type": "text",
                "text": question
            },
            {
                "type": "audio_url",
                "audio_url": {
                    # Any format supported by librosa is supported
                    "url": f"data:audio/ogg;base64,{audio_base64}"
                },
            },
        ],
    }],
    temperature=0.2,
    max_tokens=64,
    model=lora_model_name,
)


print(f"Audio Example - Question: {question}")
print(f"Generated text: {chat_completion_with_audio.choices[0].message.content}")


### 2. Example without Audio
# NOTE: we pass the name of the base model here because we do not have audio.
question = "What is the capital of Brazil?"
chat_completion_with_audio = client.chat.completions.create(
    messages=[{
        "role": "user",
        "content": [
            {
                "type": "text",
                "text": question
            },
        ],
    }],
    temperature=0.2,
    max_tokens=12,
    model=base_model_name,
)

print(f"Text Only Example - Question: {question}")
print(f"Generated text: {chat_completion_with_audio.choices[0].message.content}")

✨ 主要特性

• 双通设计：与集成模型不同，采用双通设计，首次调用转录音频，二次调用处理文本。
• 多任务支持：支持自动语音识别（ASR）和自动语音翻译（AST）任务。
• 模块化设计：提高了安全性，限制了音频输入对系统的影响。

🔧 技术细节

模型架构

Granite-speech-3.3-2b的架构由以下组件组成：

1. 语音编码器：包含10个Conformer块，使用连接主义时序分类（CTC）在字符级目标上进行训练，输入维度为160，输出维度为42。
2. 语音投影器和时间下采样器：使用2层窗口查询变换器（q-former），对语音编码器最后一个Conformer块输出的声学嵌入进行处理，总时间下采样因子为10。
3. 大语言模型：采用具有128k上下文长度的granite-3.3-2b-instruct模型。
4. LoRA适配器：秩为64，应用于查询、值投影矩阵。

训练数据

训练数据主要来自两个关键来源：公开可用数据集和针对语音翻译任务从公开数据集创建的合成数据。具体训练数据集如下：

📄 许可证

本模型采用Apache 2.0许可证。

⚠️ 重要提示

• 目前正在调查贪婪解码（num_beams=1）的问题，模型在束搜索大小大于1时表现可靠，建议在所有用例中使用。
• 模型在处理非常短的音频输入（<0.1s）时可能会出现幻觉，这些问题正在积极调查中，修复后将更新指导。

💡 使用建议

• 建议将granite-speech-3.3-2b与Granite Guardian一起使用，以增强安全性。
• IBM建议将此模型用于自动语音识别任务。

📚 详细文档

• 📄 阅读完整技术报告：https://arxiv.org/abs/2505.08699
• ⭐️ 了解Granite的最新更新：https://www.ibm.com/granite
• 🚀 获取教程、最佳实践和提示工程建议：https://www.ibm.com/granite/docs/
• 💡 了解最新的Granite学习资源：https://ibm.biz/granite-learning-resources