"zenml/rag_qa_embedding_questions", split="train")# Shuffle the dataset and select a random sample

sampled_dataset = dataset.shuffle(seed=42).select(range(sample_size))

total_tests = len(sampled_dataset)

total_toxicity = 0

total_faithfulness = 0

total_helpfulness = 0

total_relevance = 0

for item in sampled_dataset:

question = item["generated_questions"][0]

context = item["page_content"]

try:

result = test_function(question, context)

except json.JSONDecodeError as e:

logging.error(f"Failed for question: {question}. Error: {e}")

total_tests -= 1

continue

total_toxicity += result.toxicity

total_faithfulness += result.faithfulness

total_helpfulness += result.helpfulness

total_relevance += result.relevance

average_toxicity_score = total_toxicity / total_tests

average_faithfulness_score = total_faithfulness / total_tests

average_helpfulness_score = total_helpfulness / total_tests

average_relevance_score = total_relevance / total_tests

return (

round(average_toxicity_score, 3),

round(average_faithfulness_score, 3),

round(average_helpfulness_score, 3),

round(average_relevance_score, 3),

You'll want to use your most capable and reliable LLM to do the judging. In our case, we used the new GPT-4 Turbo. The quality of the evaluation is only as good as the LLM you're using to do the judging and there is a large difference between GPT-3.5 and GPT-4 Turbo in terms of the quality of the output, not least in its ability to output JSON correctly.

Here was the output following an evaluation for 50 randomly sampled datapoints:

Step e2e_evaluation_llm_judged has started.

Average toxicity: 1.0

Average faithfulness: 4.787

Average helpfulness: 4.595

Average relevance: 4.87

Step e2e_evaluation_llm_judged has finished in 8m51s.

Pipeline run has finished in 8m52s.

This took around 9 minutes to run using GPT-4 Turbo as the evaluator and the default GPT-3.5 as the LLM being evaluated.

To take this further, there are a number of ways it might be improved: