p a custom alerter as described on the Feast page,and where can I find the 'How to use it?' guide?". Expected URL ending: feature-stores.

Got: ['https://docs.zenml.io/stacks-and-components/component-guide/alerters/custom',

'https://docs.zenml.io/v/docs/stacks-and-components/component-guide/alerters/custom',

'https://docs.zenml.io/v/docs/reference/how-do-i', 'https://docs.zenml.io/stacks-and-components/component-guide/alerters',

'https://docs.zenml.io/stacks-and-components/component-guide/alerters/slack']

Loading default flashrank model for language en

Default Model: ms-marco-MiniLM-L-12-v2

Loading FlashRankRanker model ms-marco-MiniLM-L-12-v2

Loading model FlashRank model ms-marco-MiniLM-L-12-v2...

Running pairwise ranking..

Step retrieval_evaluation_full_with_reranking has finished in 4m20s.

We can see here a specific example of a failure in the reranking evaluation. It's quite a good one because we can see that the question asked was actually an anomaly in the sense that the LLM has generated two questions and included its meta-discussion of the two questions it generated. Obviously this is not a representative question for the dataset, and if we saw a lot of these we might want to take some time to both understand why the LLM is generating these questions and how we can filter them out.

Visualising our reranking performance

Since ZenML can display visualizations in its dashboard, we can showcase the results of our experiments in a visual format. For example, we can plot the failure rates of the retrieval system with and without reranking to see the impact of reranking on the performance.

Our documentation explains how to set up your outputs so that they appear as visualizations in the ZenML dashboard. You can find more information here. There are lots of options, but we've chosen to plot our failure rates as a bar chart and export them as a PIL.Image object. We also plotted the other evaluation scores so as to get a quick global overview of our performance.