s or frameworks at this stage.

bad_answers table:Question Bad Words What orchestrators does ZenML support? AWS Step Functions, Flyte, Prefect, Dagster What is the default orchestrator in ZenML? Flyte, AWS Step Functions

bad_immediate_responses table:

Question Bad Words Does ZenML support the Flyte orchestrator out of the box? Yes

good_responses table:

Question Good Words What are the supported orchestrators in ZenML? Please list as many of the supported ones as possible. Kubeflow, Airflow What is the default orchestrator in ZenML? local

Each type of test then catches a specific type of mistake. For example:

class TestResult(BaseModel):

success: bool

question: str

keyword: str = ""

response: str

def test_content_for_bad_words(

item: dict, n_items_retrieved: int = 5

) -> TestResult:

question = item["question"]

bad_words = item["bad_words"]

response = process_input_with_retrieval(

question, n_items_retrieved=n_items_retrieved

for word in bad_words:

if word in response:

return TestResult(

success=False,

question=question,

keyword=word,

response=response,

return TestResult(success=True, question=question, response=response)

Here we're testing that a particular word doesn't show up in the generated response. If we find the word, then we return a failure, otherwise we return a success. This is a simple example, but you can imagine more complex tests that check for the presence of multiple words, or the presence of a word in a particular context.

We pass these custom tests into a test runner that keeps track of how many are failing and also logs those to the console when they do:

def run_tests(test_data: list, test_function: Callable) -> float:

failures = 0

total_tests = len(test_data)

for item in test_data:

test_result = test_function(item)

if not test_result.success:

logging.error(

f"Test failed for question: '{test_result.question}'. Found word: '{test_result.keyword}'. Response: '{test_result.response}'"

failures += 1

failure_rate = (failures / total_tests) * 100

logging.info(