Deepseek R1 0528 Quantized.w4a16

🚀 DeepSeek-R1-0528-quantized.w4a16

This project focuses on the quantized version of the DeepSeek-R1-0528 model. It optimizes the model through weight quantization, reducing GPU memory and disk size requirements. It can be efficiently deployed using the vLLM backend and has been evaluated on popular reasoning tasks.

🚀 Quick Start

This model can be deployed efficiently using the vLLM backend, as shown in the example below.

from vllm import LLM, SamplingParams
from transformers import AutoTokenizer
model_id = "RedHatAI/DeepSeek-R1-0528-quantized.w4a16"
number_gpus = 8
sampling_params = SamplingParams(temperature=0.6, top_p=0.95, max_tokens=256)
tokenizer = AutoTokenizer.from_pretrained(model_id)
prompt = "Give me a short introduction to large language model."
llm = LLM(model=model_id, tensor_parallel_size=number_gpus)
outputs = llm.generate(prompt, sampling_params)
generated_text = outputs[0].outputs[0].text
print(generated_text)

vLLM also supports OpenAI-compatible serving. See the documentation for more details.

✨ Features

Model Overview

• Model Architecture: DeepseekV3ForCausalLM
  • Input: Text
  • Output: Text
• Model Optimizations:
  • Activation quantization: None
  • Weight quantization: INT4
• Release Date: 05/30/2025
• Version: 1.0
• Model Developers: Red Hat (Neural Magic)

Model Optimizations

This model was obtained by quantizing weights of DeepSeek-R1-0528 to INT4 data type. This optimization reduces the number of bits used to represent weights from 8 to 4, reducing GPU memory requirements (by approximately 50%). Weight quantization also reduces disk size requirements by approximately 50%.

💻 Usage Examples

Basic Usage

from vllm import LLM, SamplingParams
from transformers import AutoTokenizer
model_id = "RedHatAI/DeepSeek-R1-0528-quantized.w4a16"
number_gpus = 8
sampling_params = SamplingParams(temperature=0.6, top_p=0.95, max_tokens=256)
tokenizer = AutoTokenizer.from_pretrained(model_id)
prompt = "Give me a short introduction to large language model."
llm = LLM(model=model_id, tensor_parallel_size=number_gpus)
outputs = llm.generate(prompt, sampling_params)
generated_text = outputs[0].outputs[0].text
print(generated_text)

📚 Documentation

Evaluation

The model was evaluated on popular reasoning tasks (AIME 2024, MATH-500, GPQA-Diamond) via LightEval. For reasoning evaluations, we estimate pass@1 based on 10 runs with different seeds, temperature=0.6, top_p=0.95 and max_new_tokens=65536.

Accuracy

📄 License

This project is licensed under the MIT license.