lete the resources when they are no longer needed?If we generate questions for all of our chunks, we can then use these question-chunk pairs to evaluate the retrieval component. We pass the generated query to the retrieval component and then we check if the URL for the original document is in the top n results.

To generate the synthetic queries we can use the following code:

from typing import List

from litellm import completion

from structures import Document

from zenml import step

LOCAL_MODEL = "ollama/mixtral"

def generate_question(chunk: str, local: bool = False) -> str:

model = LOCAL_MODEL if local else "gpt-3.5-turbo"

response = completion(

model=model,

messages=[

"content": f"This is some text from ZenML's documentation. Please generate a question that can be asked about this text: `{chunk}`",

"role": "user",

],

api_base="http://localhost:11434" if local else None,

return response.choices[0].message.content

@step

def generate_questions_from_chunks(

docs_with_embeddings: List[Document],

local: bool = False,

) -> List[Document]:

for doc in docs_with_embeddings:

doc.generated_questions = [generate_question(doc.page_content, local)]

assert all(doc.generated_questions for doc in docs_with_embeddings)

return docs_with_embeddings

As you can see, we're using litellm again as the wrapper for the API calls. This allows us to switch between using a cloud LLM API (like OpenAI's GPT3.5 or 4) and a local LLM (like a quantized version of Mistral AI's Mixtral made available with Ollama. This has a number of advantages:

you keep your costs down by using a local model

you can iterate faster by not having to wait for API calls

you can use the same code for both local and cloud models

For some tasks you'll want to use the best model your budget can afford, but for this task of question generation we're fine using a local and slightly less capable model. Even better is that it'll be much faster to generate the questions, especially using the basic setup we have here.