Skywork O1 Open PRM Qwen 2.5 1.5B

🚀 Skywork o1 Open Model Series

The Skywork o1 Open model series, developed by the Skywork team at Kunlun Inc, introduces models with o1 - like slow - thinking and reasoning capabilities, aiming to advance AI reasoning tasks.

🤗 Hugging Face • 🤖 ModelScope

✨ Features

• Innovative Reasoning: Incorporates o1 - like slow thinking and reasoning capabilities.
• Multiple Model Options: Consists of three advanced models, each with unique strengths for different reasoning tasks.
• Improved Performance: Shows significant improvements in reasoning skills on standard benchmarks.

📦 Installation

No installation steps are provided in the original README.

💻 Usage Examples

Basic Usage

For detailed instructions on using PRM, please refer to the Skywork - o1 - PRM - Inference Repository.

Advanced Usage

huggingface inference

1. Clone the Skywork PRM inference repo:

git clone https://github.com/SkyworkAI/skywork-o1-prm-inference.git
cd skywork-o1-prm-inference

2. Run PRM inference:

from transformers import AutoTokenizer
from model_utils.prm_model import PRM_MODEL
from model_utils.io_utils import prepare_input, prepare_batch_input_for_model, derive_step_rewards

prm_model_path = "/path/to/prm_model"
tokenizer = AutoTokenizer.from_pretrained(prm_model_path, trust_remote_code=True)
datas = [
    {
        "problem"  : "Janet\u2019s ducks lay 16 eggs per day. She eats three for breakfast every morning and bakes muffins for her friends every day with four. She sells the remainder at the farmers' market daily for $2 per fresh duck egg. How much in dollars does she make every day at the farmers' market?",
        "response" : "To determine how much money Janet makes every day at the farmers' market, we need to follow these steps:\n1. **Calculate the total number of eggs laid by the ducks per day.**\n   Janet's ducks lay 16 eggs per day.\n2. **Determine the number of eggs Janet uses each day.**\n   - She eats 3 eggs for breakfast every morning.\n   - She bakes muffins for her friends every day with 4 eggs.\n   So, the total number of eggs used per day is:\n   \\[\n   3 + 4 = 7 \\text{ eggs}\n   \\]\n3. **Calculate the number of eggs Janet sells at the farmers' market each day.**\n   Subtract the number of eggs used from the total number of eggs laid:\n   \\[\n   16 - 7 = 9 \\text{ eggs}\n   \\]\n4. **Determine how much money Janet makes from selling the eggs.**\n   She sells each egg for $2, so the total amount of money she makes is:\n   \\[\n   9 \\times 2 = 18 \\text{ dollars}\n   \\]\nTherefore, the amount of money Janet makes every day at the farmers' market is $\\boxed{18}$."
    },
    {
        "problem"  : "Janet\u2019s ducks lay 16 eggs per day. She eats three for breakfast every morning and bakes muffins for her friends every day with four. She sells the remainder at the farmers' market daily for $2 per fresh duck egg. How much in dollars does she make every day at the farmers' market?",
        "res"
    }
]

📚 Documentation

Introduction

We are excited to announce the release of the Skywork o1 Open model series, developed by the Skywork team at Kunlun Inc. This groundbreaking release introduces a series of models that incorporate o1 - like slow thinking and reasoning capabilities. The Skywork o1 Open model series includes three advanced models:

• [Skywork o1 Open - Llama - 3.1 - 8B](https://huggingface.co/Skywork/Skywork - o1 - Open - Llama3.1 - 8B): A robust chat model trained on Llama - 3.1 - 8B, enhanced significantly with "o1 - style" data to improve reasoning skills.
• [Skywork o1 Open - PRM - Qwen - 2.5 - 1.5B](https://huggingface.co/Skywork/Skywork - o1 - Open - PRM - Qwen2.5 - 1.5B): A specialized model designed to enhance reasoning capability through incremental process rewards, ideal for complex problem solving at a smaller scale.
• [Skywork o1 Open - PRM - Qwen - 2.5 - 7B](https://huggingface.co/Skywork/Skywork - o1 - Open - PRM - Qwen2.5 - 7B): Extends the capabilities of the 1.5B model by scaling up to handle more demanding reasoning tasks, pushing the boundaries of AI reasoning.

Different from mere reproductions of the OpenAI o1 model, the Skywork o1 Open model series not only exhibits innate thinking, planning, and reflecting capabilities in its outputs, but also shows significant improvements in reasoning skills on standard benchmarks. This series represents a strategic advancement in AI capabilities, moving a previously weaker base model towards the state - of - the - art (SOTA) in reasoning tasks.

If you are interested in the Skywork o1 Open model series, please check out the [o1 - llama - 3.1 - 8b](https://huggingface.co/Skywork/o1 - llama - 3.1 - 8b) model.

Model Information

The Skywork - o1 - Open - PRM series are trained on [Qwen2.5 - Math - 1.5B - Instruct](https://huggingface.co/Qwen/Qwen2.5 - Math - 1.5B - Instruct) and [Qwen2.5 - Math - 7B - Instruct](https://huggingface.co/Qwen/Qwen2.5 - Math - 7B - Instruct).

PRM Evaluation

Evaluation Settings

Mathematical Evaluation

We utilized the evaluation scripts from [Qwen2.5 - Math](https://github.com/QwenLM/Qwen2.5 - Math) and followed their configuration to ensure consistency. The selected datasets include GSM8K, MATH, GaoKao, CN - Middle School 24, OlympiadBench, AIME - 24, and AMC - 23. Among these, GaoKao and CN - Middle School 24 are Chinese datasets, while the remaining datasets are in English. Notably, OlympiadBench, AIME - 24, and AMC - 23 are competition - level datasets.

Code Evaluation

For code evaluation, we adopted the evaluation scripts from [Qwen2.5 - Coder](https://github.com/QwenLM/Qwen2.5 - Coder) while largely maintaining the same configuration. The selected datasets include HumanEval, MBPP, and LiveCodeBench, with LiveCodeBench specifically using the version 2024.01 - 2024 - 11. We use the latest version (0.3.1) of evalplus due to issues with tests and code sanitization in previous versions.

Evaluation Base Models

We evaluated the performance of RMs on three base models: Qwen2.5 - 7B - Instruct, Llama3.1 - 8B - Instruct, and Skywork - o1 - Open - 8B. Data sampling was conducted to verify the performance of the RMs across different models. The sampling temperature was set to 0.7 for mathematical problems and 1.0 for code - related tasks.

Compared RMs

• [Qwen2.5 - Math - RM - 72B](https://huggingface.co/Qwen/Qwen2.5 - Math - RM - 72B): An open - source ORM provided by the Qwen team.
• [OpenR - MATH - psa - PRM - 7B](https://huggingface.co/openreasoner/Math - psa): An open - source PRM from the OpenR project.
• [RLHFlow - Deepseek - Data - PRM - 8B](https://huggingface.co/RLHFlow/Llama3.1 - 8B - PRM - Deepseek - Data): An open - source PRM from the RLHFlow project.

Evaluation Metrics

• Greedy Sampling Pass@1: Uses greedy sampling for generating answers.
• Majority Voting@64: Randomly samples 64 answers.
• Best - of - N@64: Ranks 64 answers based on output values provided by the Reward Model (RM). The weighting methods differ for ORM and PRM:
  • For ORM, only the reward from the final step is used.
  • For PRM, the average reward across all steps is used for weighting.

Evaluation Results

Mathematical Evaluation

Code Evaluation

Since the compared PRMs have not been trained on code - related tasks, this section focuses solely on the performance of Skywork - o1 - Open - PRM.

🔧 Technical Details

No technical details are provided in the original README.

📄 License

The license is "other".