Phi 4 Mini Instruct Float8dq

🚀 [Phi4-mini-instruct-float8dq模型]

[Phi4-mini-instruct-float8dq模型由PyTorch團隊使用torchao進行float8動態激活和float8權重量化（每行粒度）。可直接使用該模型，或藉助vLLM進行服務部署，在H100上可減少36%的顯存使用，提速15 - 20%，且幾乎不影響準確性。]

🚀 快速開始

本項目提供了使用vLLM和Transformers進行推理、模型量化、模型評估等功能。下面將為你詳細介紹各部分的使用方法。

✨ 主要特性

• 使用torchao對Phi4-mini模型進行float8動態激活和float8權重量化（每行粒度）。
• 可直接使用量化後的模型，也可藉助vLLM進行服務部署，在H100上可減少36%的顯存使用，提速15 - 20%，且幾乎不影響準確性。

📦 安裝指南

安裝vLLM nightly版本

pip install vllm --pre --extra-index-url https://wheels.vllm.ai/nightly
pip install torchao

推理和量化所需的其他依賴

pip install git+https://github.com/huggingface/transformers@main
pip install torch
pip install accelerate

量化安裝

pip install --pre torchao --index-url https://download.pytorch.org/whl/nightly/cu126

模型評估所需依賴

需從源碼安裝lm-eval：

git clone https://github.com/EleutherAI/lm-evaluation-harness.git
cd lm-evaluation-harness
pip install -e .

推送到Hugging Face Hub所需依賴

pip install -U "huggingface_hub[cli]"

vLLM基準測試所需依賴

git clone git@github.com:vllm-project/vllm.git
VLLM_USE_PRECOMPILED=1 pip install --editable .

下載ShareGPT數據集

wget https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json

💻 使用示例

基礎用法

使用vLLM進行推理

# 安裝vllm nightly版本以獲取最新更改
pip install vllm --pre --extra-index-url https://wheels.vllm.ai/nightly
pip install torchao

from vllm import LLM, SamplingParams

# 示例提示
prompts = [
    "Hello, my name is",
    "The president of the United States is",
    "The capital of France is",
    "The future of AI is",
]
# 創建採樣參數對象
sampling_params = SamplingParams(temperature=0.8, top_p=0.95)


if __name__ == '__main__':
    # 創建LLM實例
    llm = LLM(model="pytorch/Phi-4-mini-instruct-float8dq")
    # 從提示生成文本
    # 輸出是一個RequestOutput對象列表，包含提示、生成的文本和其他信息
    outputs = llm.generate(prompts, sampling_params)
    # 打印輸出
    print("\nGenerated Outputs:\n" + "-" * 60)
    for output in outputs:
        prompt = output.prompt
        generated_text = output.outputs[0].text
        print(f"Prompt:    {prompt!r}")
        print(f"Output:    {generated_text!r}")
        print("-" * 60)

使用Transformers進行推理

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline

torch.random.manual_seed(0)

model_path = "pytorch/Phi-4-mini-instruct-float8dq"

model = AutoModelForCausalLM.from_pretrained(
    model_path,
    device_map="auto",
    torch_dtype="auto",
    trust_remote_code=True,
)
tokenizer = AutoTokenizer.from_pretrained(model_path)

messages = [
    {"role": "system", "content": "You are a helpful AI assistant."},
    {"role": "user", "content": "Can you provide ways to eat combinations of bananas and dragonfruits?"},
    {"role": "assistant", "content": "Sure! Here are some ways to eat bananas and dragonfruits together: 1. Banana and dragonfruit smoothie: Blend bananas and dragonfruits together with some milk and honey. 2. Banana and dragonfruit salad: Mix sliced bananas and dragonfruits together with some lemon juice and honey."},
    {"role": "user", "content": "What about solving an 2x + 3 = 7 equation?"},
]

pipe = pipeline(
    "text-generation",
    model=model,
    tokenizer=tokenizer,
)

generation_args = {
    "max_new_tokens": 500,
    "return_full_text": False,
    "temperature": 0.0,
    "do_sample": False,
}

output = pipe(messages, **generation_args)
print(output[0]['generated_text'])

模型量化

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, TorchAoConfig

model_id = "microsoft/Phi-4-mini-instruct"

from torchao.quantization import Float8DynamicActivationFloat8WeightConfig, PerRow
quant_config = Float8DynamicActivationFloat8WeightConfig(granularity=PerRow())
quantization_config = TorchAoConfig(quant_type=quant_config)
quantized_model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto", torch_dtype=torch.bfloat16, quantization_config=quantization_config)
tokenizer = AutoTokenizer.from_pretrained(model_id)

# 推送到Hub
USER_ID = "YOUR_USER_ID"
MODEL_NAME = model_id.split("/")[-1]
save_to = f"{USER_ID}/{MODEL_NAME}-float8dq"
quantized_model.push_to_hub(save_to, safe_serialization=False)
tokenizer.push_to_hub(save_to)

# 手動測試
prompt = "Hey, are you conscious? Can you talk to me?"
messages = [
    {
        "role": "system",
        "content": "",
    },
    {"role": "user", "content": prompt},
]
templated_prompt = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
)
print("Prompt:", prompt)
print("Templated prompt:", templated_prompt)
inputs = tokenizer(
    templated_prompt,
    return_tensors="pt",
).to("cuda")
generated_ids = quantized_model.generate(**inputs, max_new_tokens=128)
output_text = tokenizer.batch_decode(
    generated_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False
)
print("Response:", output_text[0][len(prompt):])

高級用法

使用vLLM進行服務部署

vllm serve pytorch/Phi-4-mini-instruct-float8dq --tokenizer microsoft/Phi-4-mini-instruct -O3

模型評估

# 基準模型評估
lm_eval --model hf --model_args pretrained=microsoft/Phi-4-mini-instruct --tasks hellaswag --device cuda:0 --batch_size 8

# 量化模型評估
lm_eval --model hf --model_args pretrained=pytorch/Phi-4-mini-instruct-float8dq --tasks hellaswag --device cuda:0 --batch_size 8

峰值內存使用測試

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, TorchAoConfig

# 使用 "microsoft/Phi-4-mini-instruct" 或 "pytorch/Phi-4-mini-instruct-float8dq"
model_id = "pytorch/Phi-4-mini-instruct-float8dq"
quantized_model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto", torch_dtype=torch.bfloat16)
tokenizer = AutoTokenizer.from_pretrained(model_id)

torch.cuda.reset_peak_memory_stats()

prompt = "Hey, are you conscious? Can you talk to me?"
messages = [
    {
        "role": "system",
        "content": "",
    },
    {"role": "user", "content": prompt},
]
templated_prompt = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
)
print("Prompt:", prompt)
print("Templated prompt:", templated_prompt)
inputs = tokenizer(
    templated_prompt,
    return_tensors="pt",
).to("cuda")
generated_ids = quantized_model.generate(**inputs, max_new_tokens=128)
output_text = tokenizer.batch_decode(
    generated_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False
)
print("Response:", output_text[0][len(prompt):])

mem = torch.cuda.max_memory_reserved() / 1e9
print(f"Peak Memory Usage: {mem:.02f} GB")

模型性能基準測試

# 獲取vllm源碼
git clone git@github.com:vllm-project/vllm.git
VLLM_USE_PRECOMPILED=1 pip install --editable .

# 延遲基準測試 - 基準模型
python benchmarks/benchmark_latency.py --input-len 256 --output-len 256 --model microsoft/Phi-4-mini-instruct --batch-size 1

# 延遲基準測試 - 量化模型
VLLM_DISABLE_COMPILE_CACHE=1 python benchmarks/benchmark_latency.py --input-len 256 --output-len 256 --model pytorch/Phi-4-mini-instruct-float8dq --batch-size 1

# 服務基準測試 - 基準模型
vllm serve microsoft/Phi-4-mini-instruct --tokenizer microsoft/Phi-4-mini-instruct -O3
python benchmarks/benchmark_serving.py --backend vllm --dataset-name sharegpt --tokenizer microsoft/Phi-4-mini-instruct --dataset-path ./ShareGPT_V3_unfiltered_cleaned_split.json --model microsoft/Phi-4-mini-instruct --num-prompts 1

# 服務基準測試 - 量化模型
VLLM_DISABLE_COMPILE_CACHE=1 vllm serve pytorch/Phi-4-mini-instruct-float8dq --tokenizer microsoft/Phi-4-mini-instruct -O3
python benchmarks/benchmark_serving.py --backend vllm --dataset-name sharegpt --tokenizer microsoft/Phi-4-mini-instruct --dataset-path ./ShareGPT_V3_unfiltered_cleaned_split.json --model pytorch/Phi-4-mini-instruct-float8dq --num-prompts 1

📚 詳細文檔

模型質量評估

我們使用lm-evaluation-harness對量化模型的質量進行評估。以下是基準模型和量化模型在不同基準測試中的表現：

峰值內存使用

以下是基準模型和量化模型在推理過程中的峰值內存使用情況：

模型性能

以下是基準模型和量化模型在H100機器上的性能表現：

注意事項

• 使用VLLM_DISABLE_COMPILE_CACHE=1來禁用編譯緩存，因為vLLM和torchao的編譯組合存在一些問題，預計在PyTorch 2.8中解決。
• 要將模型推送到Hugging Face Hub，需要運行huggingface-cli login並使用具有寫入權限的令牌。

📄 許可證

本項目使用MIT許可證。