Evaluating reranking performance

Evaluate the performance of your reranking model.

We've already set up an evaluation pipeline, so adding reranking evaluation is relatively straightforward. In this section, we'll explore how to evaluate the performance of your reranking model using ZenML.

Evaluating Reranking Performance

The simplest first step in evaluating the reranking model is to compare the retrieval performance before and after reranking. You can use the same metrics we discussed in the evaluation section to assess the performance of the reranking model.

If you recall, we have a hand-crafted set of queries and relevant documents that we use to evaluate the performance of our retrieval system. We also have a set that was generated by LLMs. The actual retrieval test is implemented as follows:

def perform_retrieval_evaluation(

sample_size: int, use_reranking: bool

) -> float:

"""Helper function to perform the retrieval evaluation."""

dataset = load_dataset("zenml/rag_qa_embedding_questions", split="train")

sampled_dataset = dataset.shuffle(seed=42).select(range(sample_size))

total_tests = len(sampled_dataset)

failures = 0

for item in sampled_dataset:

generated_questions = item["generated_questions"]

question = generated_questions[

]  # Assuming only one question per item

url_ending = item["filename"].split("/")[

1

]  # Extract the URL ending from the filename

# using the method above to query similar documents

# we pass in whether we want to use reranking or not

_, _, urls = query_similar_docs(question, url_ending, use_reranking)

if all(url_ending not in url for url in urls):

logging.error(

f"Failed for question: {question}. Expected URL ending: {url_ending}. Got: {urls}"

failures += 1

logging.info(f"Total tests: {total_tests}. Failures: {failures}")

failure_rate = (failures / total_tests) * 100

return round(failure_rate, 2)