Deepseek R1 0528 AWQ

🚀 DeepSeek-R1-0528-AWQ 671B

This is a 4-bit AWQ quantization of the DeepSeek-R1-0528 671B model, suitable for use with GPU nodes such as 8xA100/8xH20/8xH100 using vLLM and SGLang.

🚀 Quick Start

You can run this model on 8x H100 80GB using vLLM with the following command:

vllm serve adamo1139/DeepSeek-R1-0528-AWQ --tensor-parallel 8

If this doesn't work for you, you may need to manually specify quantization and datatype with --quantization awq_marlin and --dtype float16 respectively.

💻 Usage Examples

Basic Usage

from datasets import load_dataset
from awq import AutoAWQForCausalLM
from transformers import AutoTokenizer

model_path = '/home/ubuntu/models/DeepSeek-R1-0528-BF16'
quant_path = '/home/ubuntu/models/DeepSeek-R1-0528-AWQ'
quant_config = { "zero_point": True, "q_group_size": 64, "w_bit": 4, "version": "GEMM" }

# Load model
model = AutoAWQForCausalLM.from_pretrained(model_path, trust_remote_code=True, device_map=None)
tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)

model.quantize(
    tokenizer,
    quant_config=quant_config,
    n_parallel_calib_samples=None,
    max_calib_samples=64,
    max_calib_seq_len=1024
)

# Save quantized model
model.save_quantized(quant_path)
tokenizer.save_pretrained(quant_path)

print(f'Model is quantized and saved at "{quant_path}"')

The author used AutoAWQ 0.2.8, transformers 4.48.0, and torch 2.6.0. modeling_deepseek.py was slightly modified to get around an issue mentioned here. Quantization was done on an 8x H100 80GB node with 960GB of RAM and 800GB of swap. The author used Unsloth's BF16 version as a starting point but removed the quantization_config section from the config.json before running the AWQ quantization script. The third attempt was successful, while the other two failed due to memory overflow after over 15 hours of runtime each. The final attempt took about 18 hours to complete. The author also mentioned that they plan to conduct some evaluations to measure the impact of quantization on downstream performance, but they haven't fully decided yet.

It's the full-fat 671B model. If you don't have access to the extreme hardware needed to run it, consider running the Qwen3 8B based distilled version instead.

Paper Link👁️

✨ Features

Introduction

The DeepSeek R1 model has undergone a minor version upgrade, and the current version is DeepSeek-R1-0528. In the latest update, DeepSeek R1 has significantly improved its depth of reasoning and inference capabilities by leveraging increased computational resources and introducing algorithmic optimization mechanisms during post-training. The model has demonstrated outstanding performance across various benchmark evaluations, including mathematics, programming, and general logic. Its overall performance is now approaching that of leading models, such as O3 and Gemini 2.5 Pro.

Compared to the previous version, the upgraded model shows significant improvements in handling complex reasoning tasks. For example, in the AIME 2025 test, the model’s accuracy has increased from 70% in the previous version to 87.5% in the current version. This advancement stems from enhanced thinking depth during the reasoning process: in the AIME test set, the previous model used an average of 12K tokens per question, whereas the new version averages 23K tokens per question.

Beyond its improved reasoning capabilities, this version also offers a reduced hallucination rate, enhanced support for function calling, and a better experience for vibe coding.

Evaluation Results

For all models, the maximum generation length is set to 64K tokens. For benchmarks requiring sampling, a temperature of 0.6, a top-p value of 0.95, and 16 responses per query are used to estimate pass@1.

Note: The Agentless framework is used to evaluate model performance on SWE-Verified. Only text-only prompts are evaluated in HLE test sets. GPT-4.1 is employed to act as the user role in Tau-bench evaluation.

Meanwhile, the chain-of-thought from DeepSeek-R1-0528 was distilled to post-train Qwen3 8B Base, obtaining DeepSeek-R1-0528-Qwen3-8B. This model achieves state-of-the-art (SOTA) performance among open-source models on the AIME 2024, surpassing Qwen3 8B by +10.0% and matching the performance of Qwen3-235B-thinking.

Chat Website & API Platform

You can chat with DeepSeek-R1 on DeepSeek's official website: chat.deepseek.com, and switch on the "DeepThink" button.

An OpenAI-Compatible API is also provided at the DeepSeek Platform: platform.deepseek.com

How to Run Locally

Please visit the DeepSeek-R1 repository for more information about running DeepSeek-R1-0528 locally.

Compared to previous versions of DeepSeek-R1, the usage recommendations for DeepSeek-R1-0528 have the following changes:

1. System prompt is now supported.
2. It is not required to add "<think>\n" at the beginning of the output to force the model into thinking pattern.

The model architecture of DeepSeek-R1-0528-Qwen3-8B is identical to that of Qwen3-8B, but it shares the same tokenizer configuration as DeepSeek-R1-0528. This model can be run in the same manner as Qwen3-8B.

System Prompt

In the official DeepSeek web/app, the same system prompt with a specific date is used.

The assistant is DeepSeek-R1, created by DeepSeek.
Today is {current date}.

For example,

The assistant is DeepSeek-R1, created by DeepSeek.
Today is Monday, May 28, 2025.

Temperature

In the web and application environments, the temperature parameter $T_{model}$ is set to 0.6.

Prompts for File Uploading and Web Search

For file uploading, please follow the template to create prompts, where {file_name}, {file_content}, and {question} are arguments.

file_template = \
"""[file name]: {file_name}
[file content begin]
{file_content}
[file content end]
{question}"""

For web search, {search_results}, {cur_date}, and {question} are arguments. For Chinese queries, the following prompt is used:

search_answer_zh_template = \
'''# The following contents are the search results related to the user's message:
{search_results}
In the search results I provide to you, each result is formatted as [webpage X begin]...[webpage X end], where X represents the numerical index of each article. Please cite the context at the end of the relevant sentence when appropriate. Use the citation format [citation:X] in the corresponding part of your answer. If a sentence is derived from multiple contexts, list all relevant citation numbers, such as [citation:3][citation:5]. Be sure not to cluster all citations at the end; instead, include them in the corresponding parts of the answer.
When responding, please keep the following points in mind:
- Today is {cur_date}.
- Not all content in the search results is closely related to the user's question. You need to evaluate and filter the search results based on the question.
- For listing-type questions (e.g., listing all flight information), try to limit the answer to 10 key points and inform the user that they can refer to the search sources for complete information. Prioritize providing the most complete and relevant items in the list. Avoid mentioning content not provided in the search results unless necessary.
- For creative tasks (e.g., writing an essay), ensure that references are cited within the body of the text, such as [citation:3][citation:5], rather than only at the end of the text. You need to interpret and summarize the user's requirements, choose an appropriate format, fully utilize the search results, extract key information, and generate an answer that is insightful, creative, and professional. Extend the length of your response as much as possible, addressing each point in detail and from multiple perspectives, ensuring the content is rich and thorough.
- If the response is lengthy, structure it well and summarize it in paragraphs. If a point-by-point format is needed, try to limit it to 5 points and merge related content.
- For objective Q&A, if the answer is very brief, you may add one or two related sentences to enrich the content.
- Choose an appropriate and visually appealing format for your response based on the user's requirements and the content of the answer, ensuring strong readability.
- Your answer should synthesize information from multiple relevant webpages and avoid repeatedly citing the same webpage.
- Unless the user requests otherwise, your response should be in the same language as the user's question.
# The user's message is:
{question}'''

For English queries, the following prompt is used:

search_answer_en_template = \
'''# The following contents are the search results related to the user's message:
{search_results}
In the search results I provide to you, each result is formatted as [webpage X begin]...[webpage X end], where X represents the numerical index of each article. Please cite the context at the end of the relevant sentence when appropriate. Use the citation format [citation:X] in the corresponding part of your answer. If a sentence is derived from multiple contexts, list all relevant citation numbers, such as [citation:3][citation:5]. Be sure not to cluster all citations at the end; instead, include them in the corresponding parts of the answer.
When responding, please keep the following points in mind:
- Today is {cur_date}.
- Not all content in the search results is closely related to the user's question. You need to evaluate and filter the search results based on the question.
- For listing-type questions (e.g., listing all flight information), try to limit the answer to 10 key points and inform the user that they can refer to the search sources for complete information. Prioritize providing the most complete and relevant items in the list. Avoid mentioning content not provided in the search results unless necessary.
- For creative tasks (e.g., writing an essay), ensure that references are cited within the body of the text, such as [citation:3][citation:5], rather than only at the end of the text. You need to interpret and summarize the user's requirements, choose an appropriate format, fully utilize the search results, extract key information, and generate an answer that is insightful, creative, and professional. Extend the length of your response as much as possible, addressing each point in detail and from multiple perspectives, ensuring the content is rich and thorough.
- If the response is lengthy, structure it well and summarize it in paragraphs. If a point-by-point format is needed, try to limit it to 5 points and merge related content.
- For objective Q&A, if the answer is very brief, you may add one or two related sentences to enrich the content.
- Choose an appropriate and visually appealing format for your response based on the user's requirements and the content of the answer, ensuring strong readability.
- Your answer should synthesize information from multiple relevant webpages and avoid repeatedly citing the same webpage.
- Unless the user requests otherwise, your response should be in the same language as the user's question.
# The user's message is:
{question}'''

📄 License

This code repository is licensed under the MIT License. The use of DeepSeek-R1 models is also subject to the MIT License. The DeepSeek-R1 series (including Base and Chat) supports commercial use and distillation.

📚 Documentation

Citation

@misc{deepseekai2025deepseekr1incentivizingreasoningcapability,
      title={DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning}, 
      author={DeepSeek-AI},
      year={2025},
      eprint={2501.12948},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2501.12948}, 
}

Contact

If you have any questions, please raise an issue or contact us at service@deepseek.com.