-dimensional vector in the high-dimensional space.We can use dimensionality reduction functionality in umap and scikit-learn to represent the 384 dimensions of our embeddings in two-dimensional space. This allows us to visualize the embeddings and see how similar chunks are clustered together based on their semantic meaning and context. We can also use this visualization to identify patterns and relationships in the data that can help us improve the retrieval performance of our RAG pipeline. It's worth trying both UMAP and t-SNE to see which one works best for our use case since they both have somewhat different representations of the data and reduction algorithms, as you'll see.

from matplotlib.colors import ListedColormap

import matplotlib.pyplot as plt

import numpy as np

from sklearn.manifold import TSNE

import umap

from zenml.client import Client

artifact = Client().get_artifact_version('EMBEDDINGS_ARTIFACT_UUID_GOES_HERE')

embeddings = artifact.load()

embeddings = np.array([doc.embedding for doc in documents])

parent_sections = [doc.parent_section for doc in documents]

# Get unique parent sections

unique_parent_sections = list(set(parent_sections))

# Tol color palette

tol_colors = [

"#4477AA",

"#EE6677",

"#228833",

"#CCBB44",

"#66CCEE",

"#AA3377",

"#BBBBBB",

# Create a colormap with Tol colors

tol_colormap = ListedColormap(tol_colors)

# Assign colors to each unique parent section

section_colors = tol_colors[: len(unique_parent_sections)]

# Create a dictionary mapping parent sections to colors

section_color_dict = dict(zip(unique_parent_sections, section_colors))

# Dimensionality reduction using t-SNE

def tsne_visualization(embeddings, parent_sections):

tsne = TSNE(n_components=2, random_state=42)

embeddings_2d = tsne.fit_transform(embeddings)

plt.figure(figsize=(8, 8))

for section in unique_parent_sections:

if section in section_color_dict:

mask = [section == ps for ps in parent_sections]

plt.scatter(

embeddings_2d[mask, 0],

embeddings_2d[mask, 1],