Rootsignals Judge Llama 70B

🚀 Model Card for RootSignals-Judge-Llama-70B

Root Judge is a powerful mid - sized LLM that enables reliable and customizable LLM system evaluations. It was post - trained from Llama - 3.3 - 70B - Instruct on a high - quality, human - annotated dataset mix for pairwise preference choice judgments and multi - turn instruction following with source citing. The model weights are freely available in FP8 to facilitate cost - effective research as well as commercial use.

Root Judge's performance surpasses the Llama - 3.3 - Instruct model and similar - sized open models on Instruction following and achieves SOTA on hallucination detection compared to leading closed models, at a fraction of the cost.

🚀 Quick Start

Via Root Signals Python SDK

The model is available on our platform as part of our evaluation suite, for no additional cost.

Install our python library:

pip install root-signals

Import:

from root import RootSignals
client = RootSignals()

Create a custom evaluator powered by Root Judge:

my_custom_judge = client.evaluators.create(
    name="Political Text Evaluator",
    intent="To measure the politics - relatedness of a given text",
    predicate="Assess if a text containts political jargon or talks about politics: {{response}}",
    model="RootJudge",
)

Execute:

result = my_custom_judge.run(
    response="A defence spending target of 3% of GDP is more likely than the 5% aim pushed by US President Donald Trump, say members of the parliamentary Defence Committee."
)
print(result.score)  # normalized score between [0 - 1]
print(result.justification)  # detailed reasoning for the score

Locally

We recommend using SGLang for production use - cases together with xml tags for important sections in your prompt. While the model can run on 80GB VRAM, we recommend at least 96GB for evaluating long - context RAG inputs.

SGlang example for a single Nvidia H100 (80GB):

docker run \
   --gpus all \
   --ipc=host  \
   -p 8000:8000 \
   -v huggingface:/root/.cache/huggingface \
   --volume /etc/localtime:/etc/localtime:ro \
   -d docker.io/lmsysorg/sglang:v0.4.2-cu124-srt \
   python3 -m sglang.launch_server \
   --model-path root-signals/RootSignals-Judge-Llama-70B \
   --host 0.0.0.0 \
   --port 8000 \
   --mem-fraction-static 0.89 \
   --grammar-backend xgrammar \
   --enable-torch-compile \
   --disable-cuda-graph

We validated the model on arm64 with vLLM on Nvidia GH200 as well with max outputs up to 64k tokens:

docker run \
   --gpus all \
   --ipc=host  \
   -p 8000:8000 \
   -v huggingface:/root/.cache/huggingface \
   --volume /etc/localtime:/etc/localtime:ro \
   -d drikster80/vllm-gh200-openai:v0.6.4.post1 \
   --model root-signals/RootSignals-Judge-Llama-70B \
   --gpu-memory-utilization 0.95 \
   --max-model-len 64k \
   --block_size 16 \
   --enable_prefix_caching

Detect hallucinations from context, example uses halubench:

decompose_system_instruction = """
<TASK>
You are a fair judge that detects hallucinations and unjustified assumptions from question - document - answer triplets provided by the user. 
Always follow the instructions below and provide your reasoning and verdict in the format specified.
</TASK>

<INSTRUCTIONS>
#1. Identify key elements in the question.
#2. List all relevant facts provided in the document.
#3. Break down the answer into its component claims.
#4. For each claim in the answer:
#a. Is it explicitly supported by the document? If yes, quote the relevant part.
#b. Is it a reasonable inference from the document? If yes, explain the reasoning.
#c. Is it unsupported or contradicted by the document? If yes, explain why.
#5. Check for any information in the answer that's present in the question but not in the document.
#6. Verify that no additional information is introduced in the answer that isn't in the document or question.
#7. Assess if the answer makes any unjustified connections or assumptions.
</INSTRUCTIONS>

<OUTPUT_EXAMPLE>
{"REASONING": "Your reasoning here where you cite the instruction step by number and provide your reasoning", "VERDICT": "PASS" or "FAIL"}
</OUTPUT_EXAMPLE>
"""

decompose_prompt = """
<QUESTION>: {question} </QUESTION>
<DOCUMENT>: {document} </DOCUMENT>
<ANSWER>: {answer} </ANSWER>
""".strip()

import os
import json
import pandas as pd
from openai import OpenAI
from pprint import pprint
from pydantic import BaseModel

testset_df = pd.read_parquet("hf://datasets/PatronusAI/HaluBench/data/test-00000-of-00001.parquet")
testset_df = testset_df.sample(frac=1).reset_index(drop=True)
example_row = testset_df.iloc[0]

class DecomposeResponse(BaseModel):
    REASONING: str
    VERDICT: str

client = OpenAI(base_url="http://localhost:8000/v1")  # export a different one for e.g. sglang, openrouter, etc.

response = client.beta.chat.completions.parse(
    model="root-signals/RootSignals-Judge-Llama-70B",  # or `RootJudge` if you are using the RootSignals API
    messages=[
        {"role": "system", "content": decompose_system_instruction},
        {"role": "user", "content": decompose_prompt.format(
            question=example_row["question"], 
            document=example_row["passage"], 
            answer=example_row["answer"])},
    ],
    response_format=DecomposeResponse,
).choices[0].message.parsed

pprint(response.REASONING)
pprint(response.VERDICT)

> ('Following the instructions: #1, the key element in the question is the '
 "nationality of the magazines. #2, the document states that 'The Woman's "
 "Viewpoint was a woman's magazine founded in Texas in 1923' and 'Pick Me Up! "
 "is a British weekly women's magazine'. #3, the answer claims both magazines "
 'are British. #4, checking each claim in the answer: a) The document does not '
 "support the claim that The Woman's Viewpoint is British, instead, it says "
 "the magazine was founded in Texas. b) There's no reasonable inference from "
 "the document that would suggest The Woman's Viewpoint is British. c) The "
 "claim about The Woman's Viewpoint is contradicted by the document. #5, the "
 'answer introduces information (both being British) not supported by the '
 'document. #6, additional information about both magazines being British is '
 'introduced in the answer without being present in the document or question. '
 '#7, the answer makes an unjustified assumption by stating both magazines are '
 "British despite the document clearly stating The Woman's Viewpoint was "
 'founded in Texas, implying it is not British. Therefore, the answer fails to '
 'accurately reflect the information provided in the document and makes '
 'unjustified assumptions based on the information given in the question and '
 "document.', ")
'FAIL'

✨ Features

Intended Use Cases

Root Judge is primarily intended to be used as an LLM - as - a - Judge in various contexts such as:

• Detecting context - grounded hallucinations, e.g. for Retrieval Augmented Generation (RAG) settings in an explainable manner, providing a justification for the score
• Pairwise preference judgments due to strong evaluation instruction - following capabilities
• Serving as a custom evaluation metric powered by use case specific evaluation rubrics
• Assisting inference - time search or synthetic data tasks that require Best - of - N decisions
• Privacy - focused settings that require local deployments

Performance Summary

Root Judge outperforms leading closed models when detecting instruction - following failures on evaluations while providing detailed, structured justifications on long inputs of up to 32k tokens on internal benchmarks and halubench public.

Hallucination Detection (in RAG setting)

📊 Benchmark: HaluBench Test Set:

* = benchmarked as o1 - preview; at current o1 prices, without reasoning tokens, the cost would start at $198.74 instead. Local Costs based on lambdalabs instances at January 2025 prices.

📄 Detailed Performance Breakdown - Hallucination Detection

Instruction Following

📊 Instruction - following performance in various diverse benchmarks compared to other open - weights judge and reward models (higher is better):

[📄 Detailed Performance Breakdown | Intruction - following](https://docs.google.com/spreadsheets/d/1cTPQZbUvelSlLkqj4kO - EQXFDkw17WXKHAeGg02 - 8Qg/edit?usp=sharing)

Root Signals Internal Benchmarks

📊 Benchmark: Root Signals Internal Hallucination Detection Benchmark

 Image 1: Total pass@1 rates and consistency (delta) assessed via ensemble of leading 3rd party models.

 Image 2: Custom rubric instruction - following by high level task.

Root Judge was tested to support complex, user - defined scoring (rating) rubrics over large context sizes. It provides granular qualitative feedback and supports structured evaluation outputs as well as tool calling.

Other Benchmarks

Despite our main focus on nuanced and transparent judgement of candidate responses, we test the judge model checkpoints extensively on public and private benchmarks to avoid known issues with performance drops such as catastrophic forgetting and find that the model preserves general capabilities of Llama - 3.3 - 70B - Instruct after dynamic weights quantization, while also slightly outperforming it on public instruction - following benchmarks such as IFEval and MuSR.

📚 Documentation

Model Details

Overview

Training Details

📄 License

The license for this model is llama3.3.

📞 Contact

Links

• Root Signals Homepage
• Root Signals Platform
• Python SDK Docs
• Root Signals GitHub
• Discord

Email

• hello@rootsignals.ai