task_categories:question-answering license:apache-2.0 region:us
---
license: apache-2.0
task_categories:
- question-answering
---

# Description
This dataset contains csv's from the Lex Fridman podcast transcripts provided by [Whispering-GPT](https://huggingface.co/datasets/Whispering-GPT/lex-fridman-podcast).
I split the episode transcripts into parent and child chunks for use with RAG.
The parent chunks is size 500 and the child is size 50. The children come with embeddings using OpenAI `text-embedding-3-small` with 1024 dimensionality.

# Motivation
This was designed for use with [ParentDocumentRetriever](https://python.langchain.com/docs/modules/data_connection/retrievers/parent_document_retriever) or [LlamaIndex](https://docs.llamaindex.ai/en/stable/examples/node_postprocessor/MetadataReplacementDemo.html).
It should provide better retrievals for queries on specific details. Embedding all the chunks took a while so hopefully this saves someone time.
The size of the csv's allow for uploading with Supabase "Table Editor" because I didn't want to use postgres COPY.