Qwen2.5 VL 7B Instruct Quantized.w8a8

🚀 Qwen2.5-VL-7B-Instruct-quantized-w8a8

This is a quantized version of Qwen/Qwen2.5-VL-7B-Instruct, which offers enhanced efficiency and performance for vision - text tasks.

🚀 Quick Start

This model can be deployed efficiently using the vLLM backend. Here is a simple example to get you started:

from vllm.assets.image import ImageAsset
from vllm import LLM, SamplingParams

# prepare model
llm = LLM(
    model="neuralmagic/Qwen2.5-VL-7B-Instruct-quantized.w8a8",
    trust_remote_code=True,
    max_model_len=4096,
    max_num_seqs=2,
)

# prepare inputs
question = "What is the content of this image?"
inputs = {
    "prompt": f"<|user|>\n<|image_1|>\n{question}<|end|>\n<|assistant|>\n",
    "multi_modal_data": {
        "image": ImageAsset("cherry_blossom").pil_image.convert("RGB")
    },
}

# generate response
print("========== SAMPLE GENERATION ==============")
outputs = llm.generate(inputs, SamplingParams(temperature=0.2, max_tokens=64))
print(f"PROMPT  : {outputs[0].prompt}")
print(f"RESPONSE: {outputs[0].outputs[0].text}")
print("==========================================")

vLLM also supports OpenAI - compatible serving. See the documentation for more details.

✨ Features

• Quantization Optimization: The model uses INT8 weight quantization and INT8 activation quantization, which significantly reduces memory usage and speeds up inference.
• Multimodal Support: It can handle vision - text inputs and generate text outputs, suitable for various multimodal tasks.

📦 Installation

The installation mainly involves setting up the necessary libraries. You can install the required dependencies according to the official documentation of vLLM and transformers.

💻 Usage Examples

Basic Usage

The above quick - start example shows the basic usage of the model for vision - text question - answering tasks.

Advanced Usage

You can adjust the parameters in the SamplingParams according to your specific requirements, such as changing the temperature to control the randomness of the generated text, or adjusting the max_tokens to limit the length of the generated text.

📚 Documentation

Model Overview

• Model Architecture: Qwen/Qwen2.5 - VL - 7B - Instruct
  • Input: Vision - Text
  • Output: Text
• Model Optimizations:
  • Weight quantization: INT8
  • Activation quantization: INT8
• Release Date: 2/24/2025
• Version: 1.0
• Model Developers: Neural Magic

Model Optimizations

This model was obtained by quantizing the weights of Qwen/Qwen2.5-VL-7B-Instruct to INT8 data type, ready for inference with vLLM >= 0.5.2.

Creation

This model was created with [llm - compressor](https://github.com/vllm - project/llm - compressor) by running the following code snippet as part of a multimodal announcement blog:

import base64
from io import BytesIO
import torch
from datasets import load_dataset
from qwen_vl_utils import process_vision_info
from transformers import AutoProcessor
from llmcompressor.modifiers.quantization import GPTQModifier
from llmcompressor.transformers import oneshot
from llmcompressor.transformers.tracing import (
    TraceableQwen2_5_VLForConditionalGeneration,
)

# Load model.
model_id = "Qwen/Qwen2.5-VL-7B-Instruct"
model = TraceableQwen2_5_VLForConditionalGeneration.from_pretrained(
    model_id,
    device_map="auto",
    torch_dtype="auto",
)
processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)

# Oneshot arguments
DATASET_ID = "lmms-lab/flickr30k"
DATASET_SPLIT = {"calibration": "test[:512]"}
NUM_CALIBRATION_SAMPLES = 512
MAX_SEQUENCE_LENGTH = 2048

# Load dataset and preprocess.
ds = load_dataset(DATASET_ID, split=DATASET_SPLIT)
ds = ds.shuffle(seed=42)

dampening_frac=0.01

# Apply chat template and tokenize inputs.
def preprocess_and_tokenize(example):
    # preprocess
    buffered = BytesIO()
    example["image"].save(buffered, format="PNG")
    encoded_image = base64.b64encode(buffered.getvalue())
    encoded_image_text = encoded_image.decode("utf-8")
    base64_qwen = f"data:image;base64,{encoded_image_text}"
    messages = [
        {
            "role": "user",
            "content": [
                {"type": "image", "image": base64_qwen},
                {"type": "text", "text": "What does the image show?"},
            ],
        }
    ]
    text = processor.apply_chat_template(
        messages, tokenize=False, add_generation_prompt=True
    )
    image_inputs, video_inputs = process_vision_info(messages)

    # tokenize
    return processor(
        text=[text],
        images=image_inputs,
        videos=video_inputs,
        padding=False,
        max_length=MAX_SEQUENCE_LENGTH,
        truncation=True,
    )

ds = ds.map(preprocess_and_tokenize, remove_columns=ds["calibration"].column_names)

# Define a oneshot data collator for multimodal inputs.
def data_collator(batch):
    assert len(batch) == 1
    return {key: torch.tensor(value) for key, value in batch[0].items()}


# Recipe
recipe = [
    GPTQModifier(
        targets="Linear",
        scheme="W8A8",
        sequential_targets=["Qwen2_5_VLDecoderLayer"],
        ignore=["lm_head", "re:visual.*"],
    ),
]

SAVE_DIR==f"{model_id.split('/')[1]}-quantized.w8a8"

# Perform oneshot
oneshot(
    model=model,
    tokenizer=model_id,
    dataset=ds,
    recipe=recipe,
    max_seq_length=MAX_SEQUENCE_LENGTH,
    num_calibration_samples=NUM_CALIBRATION_SAMPLES,
    trust_remote_code_model=True,
    data_collator=data_collator,
    output_dir=SAVE_DIR
)

Evaluation

The model was evaluated using [mistral - evals](https://github.com/neuralmagic/mistral - evals) for vision - related tasks and using [lm_evaluation_harness](https://github.com/neuralmagic/lm - evaluation - harness) for select text - based benchmarks. The evaluations were conducted using the following commands:

vllm serve neuralmagic/pixtral-12b-quantized.w8a8 --tensor_parallel_size 1 --max_model_len 25000 --trust_remote_code --max_num_seqs 8 --gpu_memory_utilization 0.9 --dtype float16 --limit_mm_per_prompt image=7

python -m eval.run eval_vllm \
        --model_name neuralmagic/pixtral-12b-quantized.w8a8 \
        --url http://0.0.0.0:8000 \
        --output_dir ~/tmp \
        --eval_name <vision_task_name>

lm_eval \
  --model vllm \
  --model_args pretrained="<model_name>",dtype=auto,add_bos_token=True,max_model_len=4096,tensor_parallel_size=<n>,gpu_memory_utilization=0.8,enable_chunked_prefill=True,trust_remote_code=True \
  --tasks mmlu \
  --num_fewshot 5 \
  --batch_size auto \
  --output_path output_dir

lm_eval \
  --model vllm \
  --model_args pretrained="<model_name>",dtype=auto,max_model_len=4096,max_gen_toks=2048,max_num_seqs=128,tensor_parallel_size=<n>,gpu_memory_utilization=0.9 \
  --tasks mgsm_cot_native \
  --apply_chat_template \
  --num_fewshot 0 \
  --batch_size auto \
  --output_path output_dir

Accuracy

Inference Performance

This model achieves up to 1.56x speedup in single - stream deployment and 1.5x in multi - stream deployment, depending on hardware and use - case scenario. The following performance benchmarks were conducted with vLLM version 0.7.2, and GuideLLM.

Single - stream performance (measured with vLLM version 0.7.2)

**Use case profiles: Image Size (WxH) / prompt tokens / generation tokens

**QPD: Queries per dollar, based on on - demand cost at [Lambda Labs](https://lambdalabs.com/service/gpu - cloud) (observed on 2/18/2025).

Multi - stream asynchronous performance (measured with vLLM version 0.7.2)

**Use case profiles: Image Size (WxH) / prompt tokens / generation tokens

**QPS: Queries per second.

**QPD: Queries per dollar, based on on - demand cost at [Lambda Labs](https://lambdalabs.com/service/gpu - cloud) (observed on 2/18/2025).

📄 License

This model is licensed under the [Apache - 2.0](https://huggingface.co/datasets/choosealicense/licenses/blob/main/markdown/apache - 2.0.md) license.