h more complex frameworks.

Preprocessing the dataOnce we have loaded the documents, we can preprocess them into a form that's useful for a RAG pipeline. There are a lot of options here, depending on how complex you want to get, but to start with you can think of the 'chunk size' as one of the key parameters to think about.

Our text is currently in the form of various long strings, with each one representing a single web page. These are going to be too long to pass into our LLM, especially if we care about the speed at which we get our answers back. So the strategy here is to split our text into smaller chunks that can be processed more efficiently. There's a sweet spot between having tiny chunks, which will make it harder for our search / retrieval step to find relevant information to pass into the LLM, and having large chunks, which will make it harder for the LLM to process the text.

import logging

from typing import Annotated, List

from utils.llm_utils import split_documents

from zenml import ArtifactConfig, log_artifact_metadata, step

logging.basicConfig(level=logging.INFO)

logger = logging.getLogger(__name__)

@step(enable_cache=False)

def preprocess_documents(

documents: List[str],

) -> Annotated[List[str], ArtifactConfig(name="split_chunks")]:

"""Preprocesses a list of documents by splitting them into chunks."""

try:

log_artifact_metadata(

artifact_name="split_chunks",

metadata={

"chunk_size": 500,

"chunk_overlap": 50

},

return split_documents(

documents, chunk_size=500, chunk_overlap=50

except Exception as e:

logger.error(f"Error in preprocess_documents: {e}")

raise

It's really important to know your data to have a good intuition about what kind of chunk size might make sense. If your data is structured in such a way where you need large paragraphs to capture a particular concept, then you might want a larger chunk size. If your data is more conversational or question-and-answer based, then you might want a smaller chunk size.