Pixtral 12b Quantized.w8a8

🚀 pixtral-12b-quantized.w8a8

A quantized version of mgoin/pixtral-12b, optimized for efficient deployment with vLLM.

🚀 Quick Start

This quantized model can be deployed efficiently using the vLLM backend. Here is a simple example to get you started:

from vllm.assets.image import ImageAsset
from vllm import LLM, SamplingParams

# prepare model
llm = LLM(
    model="neuralmagic/pixtral-12b-quantized.w8a8",
    trust_remote_code=True,
    max_model_len=4096,
    max_num_seqs=2,
)

# prepare inputs
question = "What is the content of this image?"
inputs = {
    "prompt": f"<|user|>\n<|image_1|>\n{question}<|end|>\n<|assistant|>\n",
    "multi_modal_data": {
        "image": ImageAsset("cherry_blossom").pil_image.convert("RGB")
    },
}

# generate response
print("========== SAMPLE GENERATION ==============")
outputs = llm.generate(inputs, SamplingParams(temperature=0.2, max_tokens=64))
print(f"PROMPT  : {outputs[0].prompt}")
print(f"RESPONSE: {outputs[0].outputs[0].text}")
print("==========================================")

✨ Features

• Quantization Optimization: Both weight and activation are quantized to INT8, making it ready for inference with vLLM >= 0.5.2.
• Multimodal Support: Capable of handling vision-text inputs and generating text outputs.
• Efficient Deployment: Can be deployed efficiently using the vLLM backend.

📦 Installation

No specific installation steps are provided in the original README. However, you need to have vLLM and other necessary dependencies installed to use this model.

💻 Usage Examples

Basic Usage

The basic usage example is shown above in the Quick Start section.

Advanced Usage

You can refer to the vLLM documentation for more advanced deployment and usage scenarios.

📚 Documentation

Model Overview

• Model Architecture: mgoin/pixtral-12b
  • Input: Vision-Text
  • Output: Text
• Model Optimizations:
  • Weight quantization: INT8
  • Activation quantization: INT8
• Release Date: 2/24/2025
• Version: 1.0
• Model Developers: Neural Magic

This model is a quantized version of mgoin/pixtral-12b, obtained by quantizing the weights to INT8 data type, which is ready for inference with vLLM >= 0.5.2.

Deployment

This model can be deployed efficiently using the vLLM backend. vLLM also supports OpenAI-compatible serving. See the documentation for more details.

Creation

This model was created with llm-compressor by running the following code snippet as part of a multimodal announcement blog:

import requests
import torch
from PIL import Image
from transformers import AutoProcessor

from llmcompressor.modifiers.quantization import GPTQModifier
from llmcompressor.transformers import oneshot
from llmcompressor.transformers.tracing import TraceableLlavaForConditionalGeneration

# Load model.
model_id = mgoin/pixtral-12b
model = TraceableLlavaForConditionalGeneration.from_pretrained(
    model_id, device_map="auto", torch_dtype="auto"
)
processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)

# Oneshot arguments
DATASET_ID = "flickr30k"
DATASET_SPLIT = {"calibration": "test[:512]"}
NUM_CALIBRATION_SAMPLES = 512
MAX_SEQUENCE_LENGTH = 2048


# Define a oneshot data collator for multimodal inputs.
def data_collator(batch):
    assert len(batch) == 1
    return {
        "input_ids": torch.LongTensor(batch[0]["input_ids"]),
        "attention_mask": torch.tensor(batch[0]["attention_mask"]),
        "pixel_values": torch.tensor(batch[0]["pixel_values"]),
    }


# Recipe
recipe = [
    GPTQModifier(
        targets="Linear",
        scheme="W8A8",
        sequential_targets=["MistralDecoderLayer"],
        ignore=["re:.*lm_head", "re:vision_tower.*", "re:multi_modal_projector.*"],
    ),
]

SAVE_DIR=f"{model_id.split('/')[1]}-quantized.w8a8"

# Perform oneshot
oneshot(
    model=model,
    tokenizer=model_id,
    dataset=DATASET_ID,
    splits=DATASET_SPLIT,
    recipe=recipe,
    max_seq_length=MAX_SEQUENCE_LENGTH,
    num_calibration_samples=NUM_CALIBRATION_SAMPLES,
    trust_remote_code_model=True,
    data_collator=data_collator,
    output_dir=SAVE_DIR
)

Evaluation

The model was evaluated using mistral-evals for vision-related tasks and using lm_evaluation_harness for select text-based benchmarks. The evaluations were conducted using the following commands:

vllm serve neuralmagic/pixtral-12b-quantized.w8a8 --tensor_parallel_size 1 --max_model_len 25000 --trust_remote_code --max_num_seqs 8 --gpu_memory_utilization 0.9 --dtype float16 --limit_mm_per_prompt image=7

python -m eval.run eval_vllm \
        --model_name neuralmagic/pixtral-12b-quantized.w8a8 \
        --url http://0.0.0.0:8000 \
        --output_dir ~/tmp
        --eval_name <vision_task_name>

lm_eval \
  --model vllm \
  --model_args pretrained="<model_name>",dtype=auto,add_bos_token=True,max_model_len=4096,tensor_parallel_size=<n>,gpu_memory_utilization=0.8,enable_chunked_prefill=True,trust_remote_code=True \
  --tasks mmlu \
  --num_fewshot 5
  --batch_size auto \
  --output_path output_dir \

python3 codegen/generate.py \
  --model neuralmagic/pixtral-12b-quantized.w8a8 \
  --bs 16 \
  --temperature 0.2 \
  --n_samples 50 \
  --root "." \
  --dataset humaneval

python3 evalplus/sanitize.py \
  humaneval/neuralmagic/pixtral-12b-quantized.w8a8_vllm_temp_0.2

evalplus.evaluate \
  --dataset humaneval \
  --samples humaneval/neuralmagic/pixtral-12b-quantized.w8a8_vllm_temp_0.2-sanitized

Accuracy

Inference Performance

This model achieves up to 1.57x speedup in single-stream deployment and up to 1.53x speedup in multi-stream asynchronous deployment, depending on hardware and use-case scenario. The following performance benchmarks were conducted with vLLM version 0.7.2, and GuideLLM.

Single-stream performance (measured with vLLM version 0.7.2)

**Use case profiles: Image Size (WxH) / prompt tokens / generation tokens

**QPD: Queries per dollar, based on on-demand cost at Lambda Labs (observed on 2/18/2025).

Multi-stream asynchronous performance (measured with vLLM version 0.7.2)

**Use case profiles: Image Size (WxH) / prompt tokens / generation tokens

**QPS: Queries per second.

**QPD: Queries per dollar, based on on-demand cost at Lambda Labs (observed on 2/18/2025).

🔧 Technical Details

The technical details mainly include the quantization process and the evaluation methods. The quantization is achieved using llm-compressor, and the evaluation is conducted using mistral-evals and lm_evaluation_harness.

📄 License

This model is licensed under the Apache-2.0 license.