n_name}")

# Adjust the layout

plt.tight_layout()# Save the plot to a BytesIO object

buf = io.BytesIO()

plt.savefig(buf, format="png")

buf.seek(0)

image = Image.open(buf)

return image

For one of my runs of the evaluation pipeline, this looked like the following in the dashboard:

You can see that for the full retrieval evaluation we do see an improvement. Our small retrieval test, which as of writing only included five questions, showed a considerable degradation in performance. Since these were specific examples where we knew the answers, this would be something we'd want to look into to see why the reranking model was not performing as expected.

We can also see that regardless of whether reranking was performed or not, the retrieval scores aren't great. This is a good indication that we might want to look into the retrieval model itself (i.e. our embeddings) to see if we can improve its performance. This is what we'll turn to next as we explore finetuning our embeddings to improve retrieval performance.

Try it out!

To see how this works in practice, you can run the evaluation pipeline using the project code. The reranking is included as part of the pipeline, so providing you've run the main rag pipeline, you can run the evaluation pipeline to see how the reranking model is performing.

To run the evaluation pipeline, first clone the project repository:

git clone https://github.com/zenml-io/zenml-projects.git

Then navigate to the llm-complete-guide directory and follow the instructions in the README.md file to run the evaluation pipeline. (You'll have to have first run the main pipeline to generate the embeddings.)

To run the evaluation pipeline, you can use the following command:

python run.py --evaluation

This will run the evaluation pipeline and output the results to the dashboard. As always, you can inspect the progress, logs, and results in the dashboard!

PreviousImplementing reranking in ZenML

NextImprove retrieval by finetuning embeddings

Last updated 15 days ago