Meta Llama 3.3 70B Instruct AWQ INT4

🚀 Llama 3.3 70B Instruct AWQ INT4量化模型

本項目提供了Llama 3.3 70B Instruct模型的AWQ 4位量化版本，可在不同框架下快速進行推理，同時介紹了量化過程的復現方法。

🚀 快速開始

本量化模型支持在transformers、autoawq、text-generation-inference和vLLM等不同解決方案中使用。在運行Llama 3.3 70B Instruct AWQ的INT4推理時，加載模型檢查點大約需要35 GiB的VRAM，且不包括KV緩存或CUDA圖，因此需要確保有足夠的VRAM可用。

✨ 主要特性

• 多語言支持：支持英語、法語、意大利語、葡萄牙語、印地語、西班牙語、泰語和德語等多種語言。
• 量化優化：使用AutoAWQ將模型從FP16量化到INT4，減少內存佔用，提高推理速度。
• 多框架兼容：可在transformers、autoawq、text-generation-inference和vLLM等框架中使用。

📦 安裝指南

使用transformers或autoawq運行推理

pip install -q --upgrade transformers autoawq accelerate

使用text-generation-inference運行推理

pip install -q --upgrade huggingface_hub
huggingface-cli login

使用vLLM運行推理

需安裝Docker，參考安裝說明。

💻 使用示例

基礎用法

使用transformers運行推理

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, AwqConfig

model_id = "ibnzterrell/Meta-Llama-3.3-70B-Instruct-AWQ-INT4"
quantization_config = AwqConfig(
    bits=4,
    fuse_max_seq_len=512, # Note: Update this as per your use-case
    do_fuse=True,
)

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
  model_id,
  torch_dtype=torch.float16,
  low_cpu_mem_usage=True,
  device_map="auto",
  quantization_config=quantization_config
)

prompt = [
  {"role": "system", "content": "You are a helpful assistant, that responds as a pirate."},
  {"role": "user", "content": "What's Deep Learning?"},
]
inputs = tokenizer.apply_chat_template(
  prompt,
  tokenize=True,
  add_generation_prompt=True,
  return_tensors="pt",
  return_dict=True,
).to("cuda")

outputs = model.generate(**inputs, do_sample=True, max_new_tokens=256)
print(tokenizer.batch_decode(outputs[:, inputs['input_ids'].shape[1]:], skip_special_tokens=True)[0])

使用autoawq運行推理

import torch
from awq import AutoAWQForCausalLM
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "ibnzterrell/Meta-Llama-3.3-70B-Instruct-AWQ-INT4"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoAWQForCausalLM.from_pretrained(
  model_id,
  torch_dtype=torch.float16,
  low_cpu_mem_usage=True,
  device_map="auto",
)

prompt = [
  {"role": "system", "content": "You are a helpful assistant, that responds as a pirate."},
  {"role": "user", "content": "What's Deep Learning?"},
]
inputs = tokenizer.apply_chat_template(
  prompt,
  tokenize=True,
  add_generation_prompt=True,
  return_tensors="pt",
  return_dict=True,
).to("cuda")

outputs = model.generate(**inputs, do_sample=True, max_new_tokens=256)
print(tokenizer.batch_decode(outputs[:, inputs['input_ids'].shape[1]:], skip_special_tokens=True)[0])

高級用法

使用text-generation-inference運行推理

docker run --gpus all --shm-size 1g -ti -p 8080:80 \
  -v hf_cache:/data \
  -e MODEL_ID=ibnzterrell/Meta-Llama-3.3-70B-Instruct-AWQ-INT4 \
  -e NUM_SHARD=4 \
  -e QUANTIZE=awq \
  -e HF_TOKEN=$(cat ~/.cache/huggingface/token) \
  -e MAX_INPUT_LENGTH=4000 \
  -e MAX_TOTAL_TOKENS=4096 \
  ghcr.io/huggingface/text-generation-inference:2.2.0

發送請求到部署的TGI端點：

curl 0.0.0.0:8080/v1/chat/completions \
  -X POST \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "tgi",
    "messages": [
      {
        "role": "system",
        "content": "You are a helpful assistant."
      },
      {
        "role": "user",
        "content": "What is Deep Learning?"
      }
    ],
    "max_tokens": 128
  }'

使用vLLM運行推理

docker run --runtime nvidia --gpus all --ipc=host -p 8000:8000 \
  -v hf_cache:/root/.cache/huggingface \
  vllm/vllm-openai:latest \
  --model ibnzterrell/Meta-Llama-3.3-70B-Instruct-AWQ-INT4 \
  --tensor-parallel-size 4 \
  --max-model-len 4096

發送請求到部署的vLLM端點：

curl 0.0.0.0:8000/v1/chat/completions \
  -X POST \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "ibnzterrell/Meta-Llama-3.3-70B-Instruct-AWQ-INT4",
    "messages": [
      {
        "role": "system",
        "content": "You are a helpful assistant."
      },
      {
        "role": "user",
        "content": "What is Deep Learning?"
      }
    ],
    "max_tokens": 128
  }'

📚 詳細文檔

原始模型信息

Meta Llama 3.3是一個70B參數的多語言大語言模型（LLM），經過預訓練和指令調優，適用於多語言對話場景，在常見行業基準測試中表現優於許多開源和閉源聊天模型。

量化復現信息

要使用AutoAWQ對Llama 3.3 70B Instruct進行量化，需要使用至少有足夠CPU RAM（約140GiB）來容納整個模型的實例，以及具有40GiB VRAM的NVIDIA GPU。

pip install -q --upgrade transformers autoawq accelerate

量化代碼示例：

from awq import AutoAWQForCausalLM
from transformers import AutoTokenizer
import torch

# Empty Cache
torch.cuda.empty_cache()

# Memory Limits - Set this according to your hardware limits
max_memory = {0: "22GiB", 1: "22GiB", "cpu": "160GiB"}

model_path = "meta-llama/Llama-3.3-70B-Instruct"
quant_path = "ibnzterrell/Meta-Llama-3.3-70B-Instruct-AWQ-INT4"
quant_config = {
  "zero_point": True,
  "q_group_size": 128,
  "w_bit": 4,
  "version": "GEMM"
  
}

# Load model - Note: while this loads the layers into the CPU, the GPUs (and the VRAM) are still required for quantization! (Verified with nvida-smi)
model = AutoAWQForCausalLM.from_pretrained(
    model_path,
    use_cache=False,
    max_memory=max_memory,
    device_map="cpu"
)

tokenizer = AutoTokenizer.from_pretrained(model_path)

# Quantize
model.quantize(
    tokenizer,
    quant_config=quant_config
)

# Save quantized model
model.save_quantized(quant_path)
tokenizer.save_pretrained(quant_path)

print(f'Model is quantized and saved at "{quant_path}"')

🔧 技術細節

本模型使用AutoAWQ將Llama 3.3 70B Instruct模型從FP16量化到INT4，採用GEMM內核，零點位量化，組大小為128。量化過程在單個節點上進行，該節點配備了Intel Xeon CPU E5 - 2699A v4 @ 2.40GHz、256GB的RAM和2塊NVIDIA RTX 3090（每塊24GB VRAM，總共48GB VRAM）。

📄 許可證

本項目使用的模型遵循llama3.3許可證。