Granite Speech 3.3 2b

🚀 Granite-speech-3.3-2b

Granite-speech-3.3-2b是一款緊湊高效的語音語言模型，專為自動語音識別（ASR）和自動語音翻譯（AST）而設計。該模型採用雙通設計，與將語音和語言處理整合為單通的集成模型不同。首次調用時，它會將音頻文件轉錄為文本；若要使用底層的Granite語言模型處理轉錄後的文本，用戶需進行第二次調用，因為每個步驟都必須明確啟動。

🚀 快速開始

Granite語音模型在transformers庫的main分支中得到原生支持。以下是使用granite-speech-3.3-2b模型的簡單示例。

使用transformers庫

首先，確保從源代碼構建最新版本的transformers庫：

pip install https://github.com/huggingface/transformers/archive/main.zip torchaudio peft soundfile

然後運行以下代碼：

import torch
import torchaudio
from transformers import AutoProcessor, AutoModelForSpeechSeq2Seq
from huggingface_hub import hf_hub_download

device = "cuda" if torch.cuda.is_available() else "cpu"

model_name = "ibm-granite/granite-speech-3.3-2b"
speech_granite_processor = AutoProcessor.from_pretrained(
    model_name)
tokenizer = speech_granite_processor.tokenizer
speech_granite = AutoModelForSpeechSeq2Seq.from_pretrained(
    model_name).to(device)

# prepare speech and text prompt, using the appropriate prompt template

audio_path = hf_hub_download(repo_id=model_name, filename='10226_10111_000000.wav')
wav, sr = torchaudio.load(audio_path, normalize=True)
assert wav.shape[0] == 1 and sr == 16000 # mono, 16khz

# create text prompt
chat = [
    {
        "role": "system",
        "content": "Knowledge Cutoff Date: April 2024.\nToday's Date: May 2, 2025.\nYou are Granite, developed by IBM. You are a helpful AI assistant",
    },
    {
        "role": "user",
        "content": "<|audio|>can you transcribe the speech into a written format?",
    }
]

text = tokenizer.apply_chat_template(
    chat, tokenize=False, add_generation_prompt=True
)

# compute audio embeddings
model_inputs = speech_granite_processor(
    text,
    wav,
    device=device, # Computation device; returned tensors are put on CPU
    return_tensors="pt",
).to(device)
 
model_outputs = speech_granite.generate(
    **model_inputs,
    max_new_tokens=200,
    num_beams=4,
    do_sample=False,
    min_length=1,
    top_p=1.0,
    repetition_penalty=1.0,
    length_penalty=1.0,
    temperature=1.0,
    bos_token_id=tokenizer.bos_token_id,
    eos_token_id=tokenizer.eos_token_id,
    pad_token_id=tokenizer.pad_token_id,
)

# Transformers includes the input IDs in the response.
num_input_tokens = model_inputs["input_ids"].shape[-1]
new_tokens = torch.unsqueeze(model_outputs[0, num_input_tokens:], dim=0)

output_text = tokenizer.batch_decode(
    new_tokens, add_special_tokens=False, skip_special_tokens=True
)
print(f"STT output = {output_text[0].upper()}")

使用vLLM庫

首先，確保安裝最新版本的vLLM庫：

pip install vllm --upgrade

離線模式代碼

from transformers import AutoTokenizer
from vllm import LLM, SamplingParams
from vllm.assets.audio import AudioAsset
from vllm.lora.request import LoRARequest

model_id = "ibm-granite/granite-speech-3.3-2b"
tokenizer = AutoTokenizer.from_pretrained(model_id)

def get_prompt(question: str, has_audio: bool):
    """Build the input prompt to send to vLLM."""
    if has_audio:
        question = f"<|audio|>{question}"
    chat = [
        {
            "role": "user",
            "content": question
        }
    ]
    return tokenizer.apply_chat_template(chat, tokenize=False)

# NOTE - you may see warnings about multimodal lora layers being ignored;
# this is okay as the lora in this model is only applied to the LLM.
model = LLM(
    model=model_id,
    enable_lora=True,
    max_lora_rank=64,
    max_model_len=2048, # This may be needed for lower resource devices.
    limit_mm_per_prompt={"audio": 1},
)

### 1. Example with Audio [make sure to use the lora]
question = "can you transcribe the speech into a written format?"
prompt_with_audio = get_prompt(
    question=question,
    has_audio=True,
)
audio = AudioAsset("mary_had_lamb").audio_and_sample_rate

inputs = {
    "prompt": prompt_with_audio,
    "multi_modal_data": {
        "audio": audio,
    }
}

outputs = model.generate(
    inputs,
    sampling_params=SamplingParams(
        temperature=0.2,
        max_tokens=64,
    ),
    lora_request=[LoRARequest("speech", 1, model_id)]
)
print(f"Audio Example - Question: {question}")
print(f"Generated text: {outputs[0].outputs[0].text}")


### 2. Example without Audio [do NOT use the lora]
question = "What is the capital of Brazil?"
prompt = get_prompt(
    question=question,
    has_audio=False,
)

outputs = model.generate(
    {"prompt": prompt},
    sampling_params=SamplingParams(
        temperature=0.2,
        max_tokens=12,
    ),
)
print(f"Text Only Example - Question: {question}")
print(f"Generated text: {outputs[0].outputs[0].text}")

在線模式代碼

"""
Launch the vLLM server with the following command:

vllm serve ibm-granite/granite-speech-3.3-2b \
    --api-key token-abc123 \
    --max-model-len 2048 \
    --enable-lora  \
    --lora-modules speech=ibm-granite/granite-speech-3.3-2b \
    --max-lora-rank 64
"""

import base64

import requests
from openai import OpenAI

from vllm.assets.audio import AudioAsset

# Modify OpenAI's API key and API base to use vLLM's API server.
openai_api_key = "token-abc123"
openai_api_base = "http://localhost:8000/v1"

client = OpenAI(
    # defaults to os.environ.get("OPENAI_API_KEY")
    api_key=openai_api_key,
    base_url=openai_api_base,
)

base_model_name = "ibm-granite/granite-speech-3.3-2b"
lora_model_name = "speech"
# Any format supported by librosa is supported
audio_url = AudioAsset("mary_had_lamb").url

# Use base64 encoded audio in the payload
def encode_audio_base64_from_url(audio_url: str) -> str:
    """Encode an audio retrieved from a remote url to base64 format."""
    with requests.get(audio_url) as response:
        response.raise_for_status()
        result = base64.b64encode(response.content).decode('utf-8')
    return result

audio_base64 = encode_audio_base64_from_url(audio_url=audio_url)

### 1. Example with Audio
# NOTE: we pass the name of the lora model (`speech`) here because we have audio.
question = "can you transcribe the speech into a written format?"
chat_completion_with_audio = client.chat.completions.create(
    messages=[{
        "role": "user",
        "content": [
            {
                "type": "text",
                "text": question
            },
            {
                "type": "audio_url",
                "audio_url": {
                    # Any format supported by librosa is supported
                    "url": f"data:audio/ogg;base64,{audio_base64}"
                },
            },
        ],
    }],
    temperature=0.2,
    max_tokens=64,
    model=lora_model_name,
)


print(f"Audio Example - Question: {question}")
print(f"Generated text: {chat_completion_with_audio.choices[0].message.content}")


### 2. Example without Audio
# NOTE: we pass the name of the base model here because we do not have audio.
question = "What is the capital of Brazil?"
chat_completion_with_audio = client.chat.completions.create(
    messages=[{
        "role": "user",
        "content": [
            {
                "type": "text",
                "text": question
            },
        ],
    }],
    temperature=0.2,
    max_tokens=12,
    model=base_model_name,
)

print(f"Text Only Example - Question: {question}")
print(f"Generated text: {chat_completion_with_audio.choices[0].message.content}")

✨ 主要特性

• 雙通設計：與集成模型不同，採用雙通設計，首次調用轉錄音頻，二次調用處理文本。
• 多任務支持：支持自動語音識別（ASR）和自動語音翻譯（AST）任務。
• 模塊化設計：提高了安全性，限制了音頻輸入對系統的影響。

🔧 技術細節

模型架構

Granite-speech-3.3-2b的架構由以下組件組成：

1. 語音編碼器：包含10個Conformer塊，使用連接主義時序分類（CTC）在字符級目標上進行訓練，輸入維度為160，輸出維度為42。
2. 語音投影器和時間下采樣器：使用2層窗口查詢變換器（q-former），對語音編碼器最後一個Conformer塊輸出的聲學嵌入進行處理，總時間下采樣因子為10。
3. 大語言模型：採用具有128k上下文長度的granite-3.3-2b-instruct模型。
4. LoRA適配器：秩為64，應用於查詢、值投影矩陣。

訓練數據

訓練數據主要來自兩個關鍵來源：公開可用數據集和針對語音翻譯任務從公開數據集創建的合成數據。具體訓練數據集如下：

📄 許可證

本模型採用Apache 2.0許可證。

⚠️ 重要提示

• 目前正在調查貪婪解碼（num_beams=1）的問題，模型在束搜索大小大於1時表現可靠，建議在所有用例中使用。
• 模型在處理非常短的音頻輸入（<0.1s）時可能會出現幻覺，這些問題正在積極調查中，修復後將更新指導。

💡 使用建議

• 建議將granite-speech-3.3-2b與Granite Guardian一起使用，以增強安全性。
• IBM建議將此模型用於自動語音識別任務。

📚 詳細文檔

• 📄 閱讀完整技術報告：https://arxiv.org/abs/2505.08699
• ⭐️ 瞭解Granite的最新更新：https://www.ibm.com/granite
• 🚀 獲取教程、最佳實踐和提示工程建議：https://www.ibm.com/granite/docs/
• 💡 瞭解最新的Granite學習資源：https://ibm.biz/granite-learning-resources