Thinkedit Deepseek Qwen 14b

🚀 ThinkEdit-deepseek-qwen-14b

ThinkEdit is a lightweight weight-editing method that enhances the performance of reasoning models by addressing the issue of overly short reasoning.

🚀 Quick Start

This README provides information about the ThinkEdit models, including their performance results and usage instructions.

✨ Features

• Lightweight Editing: Identifies a small percentage of "short reasoning" attention heads and edits only a tiny fraction of total parameters.
• Performance Boost: Significantly improves the accuracy of reasoning models, especially on cases with short reasoning traces.

📚 Documentation

Repository Information

• Repository for: ThinkEdit-deepseek-qwen-14b
• Authors: Chung-En Sun, Ge Yan, Tsui-Wei Weng
• Paper: ThinkEdit: Interpretable Weight Editing to Mitigate Overly Short Thinking in Reasoning Models
• Code: https://github.com/Trustworthy-ML-Lab/ThinkEdit

Introduction

Reasoning-augmented models sometimes fail by generating overly short, abstract chain-of-thought (CoT) reasoning, hurting their accuracy. ThinkEdit is a lightweight weight-editing method that:

• Identifies ~2% of "short reasoning" attention heads
• Edits only ~0.1% of total parameters
• Removes the "short reasoning" direction from their output
• Boosts performance, especially on cases with short reasoning traces

Full Performance Results

1. Overall Accuracy

2. Accuracy on Short Reasoning Cases (Top 5% / 10% / 20%)

3. Reasoning Lengths (Top 5% / 10% / 20% Shortest Responses)

Usage

The usage of ThinkEdit models is exactly the same as the original deepseek-distilled models.

Citation

@misc{sun2025thinkedit,
      title={ThinkEdit: Interpretable Weight Editing to Mitigate Overly Short Thinking in Reasoning Models}, 
      author={Chung-En Sun and Ge Yan and Tsui-Wei Weng},
      year={2025},
      eprint={2503.22048},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2503.22048}, 
}

📄 License

This project is licensed under the MIT license.