# vLLM Optimization

This is the FP8 static quantization version of OpenGVLab/InternVL3-38B, optimized for high-performance inference using vLLM. It achieves approximately 2x acceleration on vision-language tasks with minimal accuracy loss.

An FP8 quantized model based on google/gemma-3-12b-it, supporting visual-text input and text output, suitable for multimodal scenarios.

INT8 quantized version of QWQ-32B, optimized by reducing the bit-width of weights and activations

This is the quantized version of openai/whisper-large-v3, employing INT4 weight quantization and FP16 activation quantization, suitable for vLLM inference.

Quantized version of Qwen/Qwen2.5-VL-3B-Instruct, supporting visual-text input and text output, with weights quantized to INT8 and activations quantized to INT8.

pixtral-12b-FP8-dynamic is a quantized version of mistral-community/pixtral-12b. By quantizing weights and activations to the FP8 data type, it reduces disk size and GPU memory requirements by approximately 50%. It is suitable for commercial and research purposes in multiple languages.

FP8 quantized version of DeepSeek-Coder-V2-Lite-Instruct, suitable for commercial and research use in English, optimized for inference efficiency.

A quantized version of Meta-Llama-3-70B-Instruct, mainly used for English business and research purposes, capable of efficiently conducting assistant-like chats.