={

"count": len(docs_urls),

},

return docs_urlsThe get_all_pages function simply crawls our documentation website and retrieves a unique set of URLs. We've limited it to only scrape the documentation relating to the most recent releases so that we're not mixing old syntax and information with the new. This is a simple way to ensure that we're only ingesting the most relevant and up-to-date information into our pipeline.

We also log the count of those URLs as metadata for the step output. This will be visible in the dashboard for extra visibility around the data that's being ingested. Of course, you can also add more complex logic to this step, such as filtering out certain URLs or adding more metadata.

Once we have our list of URLs, we use the unstructured library to load and parse the pages. This will allow us to use the text without having to worry about the details of the HTML structure and/or markup. This specifically helps us keep the text content as small as possible since we are operating in a constrained environment with LLMs.

from typing import List

from unstructured.partition.html import partition_html

from zenml import step

@step

def web_url_loader(urls: List[str]) -> List[str]:

"""Loads documents from a list of URLs."""

document_texts = []

for url in urls:

elements = partition_html(url=url)

text = "\n\n".join([str(el) for el in elements])

document_texts.append(text)

return document_texts

The previously-mentioned frameworks offer many more options when it comes to data ingestion, including the ability to load documents from a variety of sources, preprocess the text, and extract relevant features. For our purposes, though, we don't need anything too fancy. It also makes our pipeline easier to debug since we can see exactly what's being loaded and how it's being processed. You don't get that same level of visibility with more complex frameworks.

Preprocessing the data