Skywork O1 Open PRM Qwen 2.5 7B

🚀 Skywork o1 Open Model Series

The Skywork o1 Open model series, developed by the Skywork team at Kunlun Inc, brings in models with o1 - like slow - thinking and reasoning capabilities. This series aims to enhance AI's reasoning skills and move towards the state - of - the - art in related tasks.

✨ Features

• Incorporates o1 - like slow thinking and reasoning capabilities.
• Exhibits innate thinking, planning, and reflecting capabilities in outputs.
• Shows significant improvements in reasoning skills on standard benchmarks.

📦 Installation

No installation steps are provided in the original README.

💻 Usage Examples

Basic Usage

If you want to use the PRM, refer to the following steps:

1. Clone the Skywork PRM inference repository:

git clone https://github.com/SkyworkAI/skywork-o1-prm-inference.git
cd skywork-o1-prm-inference

2. Run PRM inference:

from transformers import AutoTokenizer
from model_utils.prm_model import PRM_MODEL
from model_utils.io_utils import prepare_input, prepare_batch_input_for_model, derive_step_rewards

prm_model_path = "/path/to/prm_model"
tokenizer = AutoTokenizer.from_pretrained(prm_model_path, trust_remote_code=True)
datas = [
    {
        "problem"  : "Janet\u2019s ducks lay 16 eggs per day. She eats three for breakfast every morning and bakes muffins for her friends every day with four. She sells the remainder at the farmers' market daily for $2 per fresh duck egg. How much in dollars does she make every day at the farmers' market?",
        "response" : "To determine how much money Janet makes every day at the farmers' market, we need to follow these steps:\n1. **Calculate the total number of eggs laid by the ducks per day.**\n   Janet's ducks lay 16 eggs per day.\n2. **Determine the number of eggs Janet uses each day.**\n   - She eats 3 eggs for breakfast every morning.\n   - She bakes muffins for her friends every day with 4 eggs.\n   So, the total number of eggs used per day is:\n   \\[\n   3 + 4 = 7 \\text{ eggs}\n   \\]\n3. **Calculate the number of eggs Janet sells at the farmers' market each day.**\n   Subtract the number of eggs used from the total number of eggs laid:\n   \\[\n   16 - 7 = 9 \\text{ eggs}\n   \\]\n4. **Determine how much money Janet makes from selling the eggs.**\n   She sells each egg for $2, so the total amount of money she makes is:\n   \\[\n   9 \\times 2 = 18 \\text{ dollars}\n   \\]\nTherefore, the amount of money Janet makes every day at the farmers' market is $\\boxed{18}$."
    },
    {
        "problem"  : "Janet\u2019s ducks lay 16 eggs per day. She eats three for breakfast every morning and bakes muffins for her friends every day with four. She sells the remainder at the farmers' market daily for $2 per fresh duck egg. How much in dollars does she make every day at the farmers' market?",
        "respo"
    }
]

📚 Documentation

Introduction

The Skywork o1 Open model series includes three advanced models:

• [Skywork o1 Open - Llama - 3.1 - 8B](https://huggingface.co/Skywork/Skywork - o1 - Open - Llama3.1 - 8B): A chat model based on Llama - 3.1 - 8B, enhanced with "o1 - style" data for better reasoning.
• [Skywork o1 Open - PRM - Qwen - 2.5 - 1.5B](https://huggingface.co/Skywork/Skywork - o1 - Open - PRM - Qwen2.5 - 1.5B): A specialized model for enhancing reasoning through incremental process rewards, suitable for small - scale complex problem - solving.
• [Skywork o1 Open - PRM - Qwen - 2.5 - 7B](https://huggingface.co/Skywork/Skywork - o1 - Open - PRM - Qwen2.5 - 7B): Scaled - up version of the 1.5B model for more demanding reasoning tasks.

Model Information

The Skywork - o1 - Open - PRM series are trained on [Qwen2.5 - Math - 1.5B - Instruct](https://huggingface.co/Qwen/Qwen2.5 - Math - 1.5B - Instruct) and [Qwen2.5 - Math - 7B - Instruct](https://huggingface.co/Qwen/Qwen2.5 - Math - 7B - Instruct).

PRM Evaluation

Evaluation Settings

• Mathematical Evaluation: Used evaluation scripts from [Qwen2.5 - Math](https://github.com/QwenLM/Qwen2.5 - Math) with the same configuration. Datasets include GSM8K, MATH, GaoKao, CN - Middle School 24, OlympiadBench, AIME - 24, and AMC - 23.
• Code Evaluation: Adopted evaluation scripts from [Qwen2.5 - Coder](https://github.com/QwenLM/Qwen2.5 - Coder) with similar configuration. Datasets include HumanEval, MBPP, and LiveCodeBench (version 2024.01 - 2024 - 11).

Evaluation Base Models

Evaluated RMs on three base models: Qwen2.5 - 7B - Instruct, Llama3.1 - 8B - Instruct, and Skywork - o1 - Open - 8B. Sampling temperature was set to 0.7 for math problems and 1.0 for code tasks.

Compared RMs

• [Qwen2.5 - Math - RM - 72B](https://huggingface.co/Qwen/Qwen2.5 - Math - RM - 72B): An open - source ORM by the Qwen team.
• [OpenR - MATH - psa - PRM - 7B](https://huggingface.co/openreasoner/Math - psa): An open - source PRM from the OpenR project.
• [RLHFlow - Deepseek - Data - PRM - 8B](https://huggingface.co/RLHFlow/Llama3.1 - 8B - PRM - Deepseek - Data): An open - source PRM from the RLHFlow project.

Evaluation Metrics

• Greedy Sampling Pass@1: Uses greedy sampling for answer generation.
• Majority Voting@64: Randomly samples 64 answers.
• Best - of - N@64: Ranks 64 answers based on RM output values. ORM uses only the final - step reward, while PRM uses the average reward across all steps.

Evaluation Results

Mathematical Evaluation

Code Evaluation

Quick Start

For detailed PRM usage instructions, refer to the [Skywork - o1 - PRM - Inference Repository](https://github.com/SkyworkAI/skywork - o1 - prm - inference).

📄 License

The license is "other".

🔧 Technical Details

No specific technical details are provided in the original README.

🔗 Links

• 🤗 Hugging Face
• 🤖 ModelScope