get some immediate feedback logged to the console.This functionality can then be packaged up into a ZenML step once we're happy it does what we need:

@step

def retrieval_evaluation_small() -> Annotated[float, "small_failure_rate_retrieval"]:

failure_rate = test_retrieved_docs_retrieve_best_url(question_doc_pairs)

logging.info(f"Retrieval failure rate: {failure_rate}%")

return failure_rate

We got a 20% failure rate on the first run of this test, which was a good sign that the retrieval component could be improved. We only had 5 test cases, so this was just a starting point. In reality, you'd want to keep adding more test cases to cover a wider range of scenarios. You'll discover these failure cases as you use the system more and more, so it's a good idea to keep a record of them and add them to your test suite.

You'd also want to examine the logs to see exactly which query failed. In our case, checking the logs in the ZenML dashboard, we find the following:

Failed for question: How do I generate embeddings as part of a RAG

pipeline when using ZenML?. Expected URL ending: user-guide/llmops-guide/

rag-with-zenml/embeddings-generation. Got: ['https://docs.zenml.io/user-guide/

llmops-guide/rag-with-zenml/data-ingestion', 'https://docs.zenml.io/user-guide/

llmops-guide/rag-with-zenml/understanding-rag', 'https://docs.zenml.io/v/docs/

user-guide/advanced-guide/data-management/handle-custom-data-types', 'https://docs.

zenml.io/user-guide/llmops-guide/rag-with-zenml', 'https://docs.zenml.io/v/docs/

user-guide/llmops-guide/rag-with-zenml']

We can maybe take a look at those documents to see why they were retrieved and not the one we expected. This is a good way to iteratively improve the retrieval component.

Automated evaluation using synthetic generated queries