Flow Judge V0.1

🚀 Flow Judge v0.1

Flow Judge v0.1 是一款轻量级但功能强大的 38 亿参数模型，可在多个领域对大语言模型（LLM）系统进行定制化评估。它继承了 Phi - 3.5 - mini 指令模型的架构，在保持小体积的同时能提供高质量的评估结果。尽管模型规模较小，但在保留数据集和域外基准测试中，其性能可与更大的模型相媲美。

🚀 Flow Judge | 📄 技术报告 | 💻 flow-judge

✨ 主要特性

• 可定制评估：用户能够定义自己的评估标准和评分规则，使 Flow Judge 满足特定需求，实现对 LLM 系统性能的精准评估。
• 支持多评分体系：支持三种不同的评分尺度，包括二元通过/失败评分、3 - 李克特评分和 5 - 李克特评分，可满足不同粒度的评估需求。
• 结果易于解释：生成带有 <feedback> 和 <score> 标签的结构化评估结果。其中，定性反馈会解释评分理由并指出回复中的问题部分；评分则基于评分规则，以二元、3 - 李克特或 5 - 李克特尺度返回数值分数。

📦 安装指南

文档未提供安装步骤，故跳过此章节。

💻 使用示例

基础用法

带输入的提示模板

# GOAL
Your job is to evaluate a task carried out by an AI system powered by a large language model.
You will be provided with the inputs and output of the task, as well as the evaluation criteria and scoring rubric. Your task is to evaluate the output of the AI system based on the evaluation criteria and scoring rubric provided.
# INPUT
Below are the inputs required for performing the task:
<inputs>
{INPUTS}
</inputs>

# OUTPUT
Below is the output of the task:
<output>
{OUTPUT}
</output>

# EVALUATION CRITERIA AND SCORING RUBRIC
Here are the evaluation criteria and the rubric that you need to use for evaluating the task:
<evaluation_criteria>
{EVALUATION_CRITERIA}
</evaluation_criteria>

<scoring_rubric>
{RUBRIC}
</scoring_rubric>

# INSTRUCTIONS FOR THE EVALUATION
1. Understand the task and criteria: Familiarize yourself with the task to be evaluated. Review the evaluation criteria and scoring rubric to understand the different levels of performance and the descriptions for each score.
2. Review the inputs and output: Look at the inputs provided for the task. Examine the output generated from completing the task.
3. Compare output to score descriptions: Compare the output against the criteria and score descriptions in the scoring rubric. For each criterion,decide which description best matches the output.
4. After comparing the output to the score descriptions, pay attention to the small details that might impact the final score that you assign. Sometimes a small difference can dictate the final score.
5. Write verbal feedback justifying your evaluation that includes a detailed rationale, referring to specific aspects of the output and comparing them to the rubric.
6. Assign a final score based on the scoring rubric.

## FORMAT FOR THE EVALUATION
- Write the verbal feedback inside <feedback> tags without any additional surrounding text.
- Write the numeric score inside <score> tags, without any additional surrounding text and always after the feedback.

Please accurately evaluate the task. Strictly adhere to the evaluation criteria and rubric.

不带输入的提示模板

# GOAL
Your job is to evaluate a task carried out by an AI system powered by a large language model.

You will be provided the output of the task, as well as the evaluation criteria and scoring rubric. Your task is to evaluate the output of the AI system based on the evaluation criteria and scoring rubric provided.

# OUTPUT
Below is the output of the task:
<output>
{OUTPUT}
</output>

# EVALUATION CRITERIA AND SCORING RUBRIC
Here are the evaluation criteria and the rubric that you need to use for evaluating the task:
<evaluation_criteria>
{EVALUATION_CRITERIA}
</evaluation_criteria>

<scoring_rubric>
{RUBRIC}
</scoring_rubric>

# INSTRUCTIONS FOR THE EVALUATION
1. Understand the task and criteria: Familiarize yourself with the task to be evaluated. Review the evaluation criteria and scoring rubric to understand the different levels of performance and the descriptions for each score.
2. Review the output: Examine the output generated from completing the task.
3. Compare output to score descriptions: Compare the output against the criteria and score descriptions in the scoring rubric. For each criterion,decide which description best matches the output.
4. After comparing the output to the score descriptions, pay attention to the small details that might impact the final score that you assign. Sometimes a small difference can dictate the final score.
5. Write verbal feedback justifying your evaluation that includes a detailed rationale, referring to specific aspects of the output and comparing them to the rubric.
6. Assign a final score based on the scoring rubric.

## FORMAT FOR THE EVALUATION
- Write the verbal feedback inside <feedback> tags without any additional surrounding text.
- Write the numeric score inside <score> tags, without any additional surrounding text and always after the feedback.

Please accurately evaluate the task. Strictly adhere to the evaluation criteria and rubric.

带输入的格式化提示示例

# GOAL
Your job is to evaluate a task carried out by an AI system powered by a large language model.
You will be provided with the inputs and output of the task, as well as the evaluation criteria and scoring rubric. Your task is to evaluate the output of the AI system based on the evaluation criteria and scoring rubric provided.
# INPUT/s
Below are the inputs required for performing the task:
<inputs>
<customer_query>
Subject: Issues with recent order #12345
Hello,
I recently received my order #12345, but I'm having a few problems:
1. The blue t-shirt I ordered is actually green. Can I exchange it?
2. The phone case doesn't fit my iPhone 13 Pro Max. The website said it was compatible.
3. I was charged for express shipping, but it took over a week to arrive. Can I get a refund on the shipping cost?
Please let me know how we can resolve these issues. I've been a loyal customer for years and this is the first time I've had problems like this.
Thank you,
Jane Smith
</customer_query>
</inputs>

# OUTPUT
Below is the output of the task:
<output>
Dear Jane Smith,
Thank you for reaching out about the issues with your recent order #12345. We sincerely apologize for the inconvenience you've experienced. Let me address each of your concerns:
1. T-shirt color discrepancy: We're sorry about the mix-up with the t-shirt color. We'd be happy to exchange it for the correct blue t-shirt. I'll send you a prepaid return label for the green shirt, and we'll ship out the blue one as soon as we receive the return.
2. Phone case compatibility: We apologize for the confusion regarding the phone case compatibility. Our website information should have been more accurate. We'll gladly accept a return of the incompatible case and issue a full refund for it. Additionally, I can recommend a case that will fit your iPhone 13 Pro Max if you're interested.
3. Shipping charge: You're absolutely right about the shipping charge. Since the delivery didn't meet our express shipping standards, we will refund the entire shipping cost to your original payment method. This refund should process within 3-5 business days.
To initiate these resolutions, please reply to this email confirming that you'd like to proceed with the t-shirt exchange and phone case return. Once confirmed, I'll send you the necessary return labels and process the shipping refund immediately.
We truly value your loyalty as a long-time customer and sincerely regret that you've encountered these issues. Rest assured, we're taking steps to prevent similar problems in the future. As a gesture of goodwill, we'd like to offer you a 20% discount on your next order.
If you have any further questions or concerns, please don't hesitate to reach out. We're here to ensure your complete satisfaction.
Best regards,
Alex Johnson
Customer Service Representative
</output>

# EVALUATION CRITERIA AND SCORING RUBRIC
Here are the evaluation criteria and the rubric that you need to use for evaluating the task:
<evaluation_criteria>
How well the response addresses the specific issues raised in the customer's query?
</evaluation_criteria>
<scoring_rubric>
- Score 1: The response completely fails to address the customer's needs and ignores the specific issues raised.
- Score 2: The response barely addresses the customer's query and misses most of the specific issues raised.
- Score 3: The response partially addresses the customer's query, touching on some of the specific issues but leaving others unaddressed.
- Score 4: The response adequately addresses most aspects of the customer's query and the specific issues raised.
- Score 5: The response fully and comprehensively addresses all aspects of the customer's query and all specific issues raised in a highly satisfactory manner.
</scoring_rubric>

# INSTRUCTIONS FOR THE EVALUATION
1. Understand the task and criteria: Familiarize yourself with the task to be evaluated. Review the evaluation criteria and scoring rubric to understand the different levels of performance and the descriptions for each score.
2. Review the inputs and output: Look at the inputs provided for the task. Examine the output generated from completing the task.
3. Compare output to score descriptions: Compare the output against the criteria and score descriptions in the scoring rubric. For each criterion,decide which description best matches the output.
4. After comparing the output to the score descriptions, pay attention to the small details that might impact the final score that you assign. Sometimes a small difference can dictate the final score.
5. Write verbal feedback justifying your evaluation that includes a detailed rationale, referring to specific aspects of the output and comparing them to the rubric.
6. Assign a final score based on the scoring rubric.

## FORMAT FOR THE EVALUATION
- Write the verbal feedback inside <feedback> tags without any additional surrounding text.
- Write the numeric score inside <score> tags, without any additional surrounding text and always after the feedback.

Please accurately evaluate the task. Strictly adhere to the evaluation criteria and rubric.

📚 详细文档

预期用例

Flow Judge 旨在用于定制化的 LLM 系统评估任务，具体如下：

• 可定制评估：用户可定义自己的评估标准和评分规则，使评估更具针对性，准确衡量 LLM 系统的性能。
• 多评分体系支持：
  • 二元通过/失败评分：适用于二元评估，如判断一段文本是否符合特定标准或是否包含错误。
  • 3 - 李克特评分：允许更细致的评估，分数范围从负面到中立再到正面，有助于评估文本的整体质量或情感倾向。
  • 5 - 李克特评分：提供更细致的评估，分数范围从强烈负面到强烈正面，能捕捉质量或情感的细微差异。
• 结果易于解释：Flow Judge 生成带有 <feedback> 和 <score> 标签的结构化评估结果。定性反馈会解释评分理由并指出回复中的问题部分；评分则基于评分规则，以二元、3 - 李克特或 5 - 李克特尺度返回数值分数。

量化权重

• flowaicom/Flow - Judge - v0.1 - AWQ
• flowaicom/Flow - Judge - v0.1 - GGUF

快速开始

• 快速开始

🔧 技术细节

模型

Flow Judge 基于 Phi - 3.5 - mini 架构，具体使用其指令版本的基础模型检查点。该模型使用相同的分词器，支持多查询注意力（MQA）和 Flash Attention 2，权重采用 bfloat16 精度。不过，微调后模型对语言和长上下文长度的支持尚未完全测试。由于经过专门的监督微调（SFT），Flow Judge 可能会显示不同的基准测试结果，并且最大上下文长度为 8192，比基础模型短。

训练数据集

Flow - Judge - v0.1 在合成生成的数据集上进行训练。其训练数据集的构建包括以下多步骤过程：

1. 手动策划种子评分规则作为基础。
2. 为不同领域合成生成适应特定领域的指标和评分规则。
3. 合成生成包含多个输入（如用户查询和上下文信息）的训练实例。
4. 采用双重评估策略并达成共识，以确保质量和一致性。

此过程创建了全面且多样化的训练实例集，可在生成式 AI 产品中对 LLM 系统进行准确的特定领域评估，同时减少人工干预。更多关于数据集构建的信息可查看此处。

微调

微调时使用 Axolotl 的预处理方法确保输入训练数据的一致性。然后基于 microsoft/Phi - 3.5 - mini - instruct 进行监督微调，采用 RSLoRa 方法。更多关于微调过程的详细信息可查看技术报告。

📄 许可证

本项目采用 Apache 2.0 许可证，详情请见 LICENSE。