NLP for taking a string of text and identifying duplications and building a new string without the duplication
Hey there,I’ve been working on something that uses GPT to clean up large audio recordings and transcripts.I basically send a transcript through GPT in chunks which formats it and cleans up the input. However since the recordings are quite large, there’s only so many tokens I can send through at once. So I’m I’ve tried breaking down the string, sending it through GPT and then reassembling it afterwards.However once I reassemble, there’s obvious overlaps.Any tips on how I can get clean up the overlapping text?See example below:We can lie with our words, but we cannot lie with that which we don't know exists. Therefore, truth for all of us resides in the unavoidable sounds we make. This even happens when we're saying something completely scripted, when we know every word we're going to say. 

Like, have you ever been told a joke by a nine-year-old kid? It's excruciating - it's kind of cute, but it's a horrible, horrible experience. The reason for this is kids are like little approval-guided missiles, zeroing in on anywhere they think they can hit a little piece of validation. So, when they tell a joke, even if they know every word, we hear their need instead of the punchline. 

        ** Insecurities will always make sounds that bleed out between our words. So, even when we're saying something exceedingly simple, like a single sentence, we still make noises that reveal more than just our words.

        We can lie with our words, but we cannot lie with that which we don't know exists, therefore truth for all of us resides in the unavoidable sounds we make. This even happens when we're saying something completely scripted, when we know every word we're going to say. 

        Have you ever been told a joke by a nine-year-old kid? It's excruciating; it's kind of cute, but it's a horrible, horrible experience. The reason for this is kids are like little approval-guided missiles, zeroing in on any place they think they can hit a little piece of validation. So when they tell a joke, even if they know every word, we hear their need instead of the punchline**. 

Insecurities will always make sounds that bleed out between our words, so even when we're saying something exceedingly simple, like a single sentence, this kind of thing happens all the time. Like has anyone ever said something like this to you: "Honey, I love you..." That really long 'you' at the end is called a leading tone. What's this person actually saying? The high-level tone at the end of a sentence is a prompt; it's actually asking a question, so they're not saying 'I love you' - they're asking 'Do you love me?' 

So now that we have some idea that we're constantly giving off these vocal cues, that must give us the ability to begin to manage them, right? Yeah, a little bit. There are lots and lots of vocal cues - to date, we've catalogued over 1200 neural-mechanical reactions to stress or vocal cues. So for most of us, vocal cues simply come too quickly and occur too often to provide us with any sense of control, even when we go well out of our way to manipulate them.