Phi3 Rag Relevance Judge Merge

🚀 Merged Model Performance

This repository holds the results of our merged rag relevance PEFT adapter model, which aims to provide efficient solutions for binary classification tasks related to RAG relevance.

🚀 Quick Start

This section provides an overview of the model's performance, usage, and comparison with other state - of - the - art language models.

✨ Features

RAG Relevance Classification Metrics

Our merged model achieves the following performance on a binary classification task:

              precision    recall  f1-score   support

           0       0.74      0.77      0.75       100
           1       0.76      0.73      0.74       100

    accuracy                           0.75       200
   macro avg       0.75      0.75      0.75       200
weighted avg       0.75      0.75      0.75       200

Model Usage

For best results, we recommend starting with the following prompting strategy (and encourage tweaks as you see fit):

def format_input_classification(query, text):
    input = f"""
      You are comparing a reference text to a question and trying to determine if the reference text
  contains information relevant to answering the question. Here is the data:
      [BEGIN DATA]
      ************
      [Question]: {query}
      ************
      [Reference text]: {text}
      ************
      [END DATA]
  Compare the Question above to the Reference text. You must determine whether the Reference text
  contains information that can answer the Question. Please focus on whether the very specific
  question can be answered by the information in the Reference text.
  Your response must be single word, either "relevant" or "unrelated",
  and should not contain any text or characters aside from that word.
  "unrelated" means that the reference text does not contain an answer to the Question.
  "relevant" means the reference text contains an answer to the Question."""
    return input


text = format_input_classification("What is quanitzation?",
  "Quantization is a method to reduce the memory footprint")
messages = [
    {"role": "user", "content": text}
]

pipe = pipeline(
    "text-generation",
    model=base_model,
    model_kwargs={"attn_implementation": attn_implementation, "torch_dtype": torch.float16},
    tokenizer=tokenizer,
)

Comparison with Other Models

We compared our merged model's performance on the RAG Eval benchmark against several other state - of - the - art language models:

[1] Scores from arize/phoenix

As shown in the table, our merged model achieves a comparable score of 0.75, outperforming several other black box models.

We will continue to improve and fine - tune our merged model to achieve even better performance across various benchmarks and tasks.

📚 Documentation

Citations

[1] https://docs.arize.com/phoenix/evaluation/how - to - evals/running - pre - tested - evals/retrieval - rag - relevance