Spotify's Podcast Search Explained

OK, so you're looking at 700 plus dimensions in these vector representations.


Spotify's Podcast Search Explained

In this one, we've used PCA to reduce the dimensionality of those very high dimensional vectors into a 3D space,


Spotify's Podcast Search Explained

just so we can visualize it, but still maintain the sort of relationships between those vectors.


Spotify's Podcast Search Explained

So if we have a look here, we can see there's one main grouping of vectors.


Spotify's Podcast Search Explained

OK, so these three over here.


Spotify's Podcast Search Explained

Now, let's just zoom in a little bit and have a look what they are.


Spotify's Podcast Search Explained

So we have text, podcasts about cooking and writing, a chat with chef who wrote books, an interview with cookbook author.


Spotify's Podcast Search Explained

So we can see that they're all about cooking and writing and they've all been grouped together.


Spotify's Podcast Search Explained

Now, that's because our language model has taken these queries and encoded them in such a way that the vector representations for these represent the meaning.


Spotify's Podcast Search Explained

OK, and the meaning of all these, although it's not the same, is very similar, talking about cooking, writing books and so on.


Spotify's Podcast Search Explained

And we can see that represented in the proximity of these vectors.


Spotify's Podcast Search Explained

Now, let's have a look at something else that's kind of close, but not within that same cluster.


Spotify's Podcast Search Explained

So over here we have how do I cook great food? OK, so we're talking about cooking again.


Spotify's Podcast Search Explained

OK, and then we have this one further over here, eat better during Christmas holidays.


Spotify's Podcast Search Explained

So we can kind of see that this one is how do I cook great food?


Spotify's Podcast Search Explained

It's sort of in between like these about cooking and writing and a cookbook.


Spotify's Podcast Search Explained

And then over here we have eat better during Christmas holidays.


Spotify's Podcast Search Explained

I was talking specifically about food. OK, so this one is cooking and food and it's in the middle of them both.


Spotify's Podcast Search Explained

And then we have a few more over here. We have the writing show over here, so it's like a writing podcast or something.


Spotify's Podcast Search Explained

And then over here, one of the more similar vectors is how to tell more engaging stories.


Spotify's Podcast Search Explained

And then also how to keep readers interested. So again, talking about writing.


Spotify's Podcast Search Explained

And then we go up here, we see this is by itself superhero film and arts.


Spotify's Podcast Search Explained

It's obviously very different to everything else we've looked at.


Spotify's Podcast Search Explained

So this is kind of what I mean by representing the meaning numerically in a vector space.


Spotify's Podcast Search Explained

Similar things get grouped together. Dissimilar things are separated.


Spotify's Podcast Search Explained

Now, all of those vectors have been encoded by a special kind of NLP model called a sentence transformer.


Spotify's Podcast Search Explained

Now, that sentence transformer is specially trained to do what we just saw, which is group similar things together, separate dissimilar things.


Spotify's Podcast Search Explained

And once we have those vectors from our sentence transformer, we need a way to compare them.


Spotify's Podcast Search Explained

And that's why the vector search component comes in.


Spotify's Podcast Search Explained

So imagine within all those queries that we just saw, imagine we came up with a new query.


Spotify's Podcast Search Explained

Convert that query into a vector, place it within that vector space, and then we would search for the most similar other vectors.


Spotify's Podcast Search Explained

And we would return those as being the ones that are most similar to your particular query.


Spotify's Podcast Search Explained

Now, NLP and vector search have been around for a long time.


Spotify's Podcast Search Explained

Both fields have had very recent developments that have really acted as catalysts for the performance increases and subsequent adoption of semantic search.


Spotify's Podcast Search Explained

In NLP, we've had the introduction of transformer models and in vector search, the introduction of ever better, approximate, nearest neighbor search algorithms.


Spotify's Podcast Search Explained

Now, transformers and approximate nearest neighbors have powered the growth of semantic search, but it's not inherently clear what both of those are.


Spotify's Podcast Search Explained

I'm assuming most of you probably have no idea what either of those are. If you do, then that's great.


Spotify's Podcast Search Explained

But otherwise, no problem, because we're going to actually go through both of those concepts as well.


Spotify's Podcast Search Explained

So starting with transformer models, transformer models are essentially the standard in NLP now.


Spotify's Podcast Search Explained

They typically consist of two components, which is quite important for us to know.


Spotify's Podcast Search Explained

There's the core of the transformer, and then you usually have a head, which adapts the transformer for a particular task.


Spotify's Podcast Search Explained

Now, there's just one problem. The core of these transformer models is huge.


Spotify's Podcast Search Explained

Like, they're massive models, and for most organizations, it is far too expensive to actually train the core of a model.


Spotify's Podcast Search Explained

For example, with BERT, which is one of the most popular language models, it's not even a particularly large language model.


Spotify's Podcast Search Explained

That costs, we reported, I think, two and a half thousand to fifty thousand dollars to train a small one.


Spotify's Podcast Search Explained

And then when you look at the larger BERT model, that shifts up to eighty thousand to one point six million dollars, which is obviously huge.


Spotify's Podcast Search Explained

Now, most organizations don't have that kind of money lying around.


Spotify's Podcast Search Explained

So how do we, you know, how are transformers useful for us if we can't afford to train them?


Spotify's Podcast Search Explained

Well, the way that we typically go about using transformer models is, one,


Spotify's Podcast Search Explained

the core of a transformer model is trained by the likes of Microsoft or Google.


Spotify's Podcast Search Explained

It costs them a lot of money to train it, but they have the money to do that.


Spotify's Podcast Search Explained

Two, the model is made publicly available by Microsoft or Google.


Spotify's Podcast Search Explained

Three, other organizations take this core, this transformer core, and they add different heads to it.


Spotify's Podcast Search Explained

So these heads are like a final few layers at the end of the model that adapt it for a particular task.


Spotify's Podcast Search Explained

Now, using that extended model with the head, the model can be fine tuned using a lot less computing power,


Spotify's Podcast Search Explained

which brings it within the range of feasibility for other more normal organizations.


Spotify's Podcast Search Explained

And four, once you've fine tuned your extended transformer model,


Spotify's Podcast Search Explained

you are able to go ahead and actually apply that to your specific task.


Spotify's Podcast Search Explained

So in the case of our example, Podcast Search, we might want to take a BERT model.


Spotify's Podcast Search Explained

So like BERT based on CASE, which has been pre-trained by Google.


Spotify's Podcast Search Explained

We will take that and then we would add a what's called a pooling head onto the end of it.


Spotify's Podcast Search Explained

That converts it into what we call sentence transformers, which will take an input like a normal transformer,


Spotify's Podcast Search Explained

so some sort of text input. The normal transformer will output a set of token level embeddings.


Spotify's Podcast Search Explained

Now, we can't use token level embeddings when we're comparing sentences because one token or one sentence


Spotify's Podcast Search Explained

for BERT can contain up to 512 tokens.


Spotify's Podcast Search Explained

So we really need a way to actually compress that down into one single vector to represent the full set of inputs


Spotify's Podcast Search Explained

that we've given to our transformer model. So that's where we have that pooling layer.


Spotify's Podcast Search Explained

That pooling layer takes all of those token level embeddings and converts them into a single embedding,


Spotify's Podcast Search Explained

which is our sentence vector or sentence embedding.


Spotify's Podcast Search Explained

Now, there are different types of pooling layers, and we'll talk about one later.


Spotify's Podcast Search Explained

But they all consume the same input and they will output that single embedding.


Spotify's Podcast Search Explained

And it is that sort of model that we refer to as a sentence transformer.


Spotify's Podcast Search Explained

That sentence embedding at the end is what we referred to earlier as a dense vector.


Spotify's Podcast Search Explained

The numerical representation of whatever we fed into that transformer model.


Spotify's Podcast Search Explained

So given that dense vector, we move on to the next step, which is a vector search.


Spotify's Podcast Search Explained

Now, approximate nearest neighbors search allows us to search more efficiently through a lot of vectors


Spotify's Podcast Search Explained

because in many use cases, for example, with Spotify, how many episodes do they have on that?


Spotify's Podcast Search Explained

I imagine there are many. And it's very hard if you or it takes a very long time


Spotify's Podcast Search Explained

if you compare your query to all of the existing vector embeddings.


Spotify's Podcast Search Explained

It's very hard to do that quickly.


Spotify's Podcast Search Explained

In fact, it's impossible if you're using a typical k nearest neighbors search


Spotify's Podcast Search Explained

or otherwise known as exhaustive search because you need to compare everything.


Spotify's Podcast Search Explained

And if you have millions of items to compare there, no matter what hardware you run, it's going to take a long time.


Spotify's Podcast Search Explained

So approximate search allows us to speed that up by literally approximating the answer.


Spotify's Podcast Search Explained

Now, approximate nearest neighbor algorithms are very good because they allow you to approximate the answer


Spotify's Podcast Search Explained

with a super high degree of accuracy, like 99 percent accuracy, and make your search incredibly fast.


Spotify's Podcast Search Explained

You're talking sort of sub second, sub half a second. You're in the 10 millisecond range in a lot of cases.


Spotify's Podcast Search Explained

So that's really useful. But it's also worth noting that there is between these different algorithms,


Spotify's Podcast Search Explained

usually a trade off between speed, like your search times and the accuracy.


Spotify's Podcast Search Explained

So you can have an even faster search, but then your accuracy will tend to be not quite as good.


Spotify's Podcast Search Explained

So you'll be returning maybe 70 percent accuracy rather than 99 percent accuracy.


Spotify's Podcast Search Explained

But there's always sort of a trade off between the two.


Spotify's Podcast Search Explained

But generally the algorithm is now very good and you can get incredible accuracy and incredible speed as well.


Spotify's Podcast Search Explained

So we've merged those two together. We have our sentence transformers.


Spotify's Podcast Search Explained

We have our approximate search. And we now have a tool set that is capable of performing semantic search at incredibly large scales.


Spotify's Podcast Search Explained

So what I want to do now is look specifically at how Spotify did it.


Spotify's Podcast Search Explained

To build this kind of semantic search tool, Spotify first needed a model that can encode queries and episode metadata or descriptions into the same sort of vector space.


Spotify's Podcast Search Explained

Now, there are existing transform models or sentence transform models that can do a lot of things.


Spotify's Podcast Search Explained

For example, Espert. But Spotify found a few issues with those.


Spotify's Podcast Search Explained

Specifically for Espert, they needed a model that was capable of supporting multilingual queries because obviously Spotify has a lot of content from every place in the world.


Spotify's Podcast Search Explained

So they couldn't use Espert because Espert has been trained on English only data.


Spotify's Podcast Search Explained

And two, Espert's performance on new domains, so for example, podcasts, is not great without further fine tuning.


Spotify's Podcast Search Explained

So the out of the box, pre-trained, fine tuned sentence model could not be used. It didn't satisfy what Spotify needed.


Spotify's Podcast Search Explained

With that in mind, they decided to start with the pre-trained universal sentence encoder model.


Spotify's Podcast Search Explained

This allowed them to convert a multilingual issue. And yes, the USC model still needs to be fine tuned, but they're going to need to do that anyway.


Spotify's Podcast Search Explained

So it's not really an issue. So let's have a look at what sort of data they use to actually fine tune their model.


Spotify's Podcast Search Explained

So to fine tune their model, Spotify needs query, episode pairs.


Spotify's Podcast Search Explained

Now, of course, Spotify has a ton of search logs over here.


Spotify's Podcast Search Explained

So they took these search logs and they use them to create two different data sources.


Spotify's Podcast Search Explained

The first was simply, OK, if there is a successful search, what was the query from that search and what was the episode?


Spotify's Podcast Search Explained

OK, and that created this data source that you see here. The other one's a bit more interesting.


Spotify's Podcast Search Explained

So whenever they found that there was a unsuccessful search that was straightaway followed by a successful search,


Spotify's Podcast Search Explained

they looked at what the query for that unsuccessful search was because the logic behind that being if you're typing something, you're searching and it doesn't work,


Spotify's Podcast Search Explained

you probably use a more sort of natural language or a more natural feeling query.


Spotify's Podcast Search Explained

But then it doesn't work. So then you change it to be a bit more robotic and trying to fit what you would expect Spotify's search to come up with


Spotify's Podcast Search Explained

for the correct episode or podcast episode. So what they did is they took that unsuccessful search query


Spotify's Podcast Search Explained

and then the episode that the user found after their successful search and put them together to create this little data source.


Spotify's Podcast Search Explained

So we have two data sources. Now, we use dotted lines here because we can't replicate that. We don't have Spotify's past search logs.


Spotify's Podcast Search Explained

So we will just ignore that. But we'll be aware that the way that they would be used would be exactly the same as this third data source over here,


Spotify's Podcast Search Explained

which we are going to replicate. Now that third data source is taking podcast episodes. So it's a little diamond over here.


Spotify's Podcast Search Explained

We're transforming them. So what I mean by transforming is taking it like the podcast show title and description


Spotify's Podcast Search Explained

and also the specific episode title and description and concatenating all that together to create a sort of episode description from these multiple things.


Spotify's Podcast Search Explained

And then we use those episodes, that episode data with a query generation set. Now query generation is super interesting.


Spotify's Podcast Search Explained

And what we're going to do there is actually use a query generation model to create synthetic queries for our episodes.


Spotify's Podcast Search Explained

And that will create this third data source over here, which is synthetic queries to episode pairs.


Spotify's Podcast Search Explained

And we'll be using that. So we'll split that into sort of a training set and we'll use that to fine tune a model in the same way that we would have used


Spotify's Podcast Search Explained

the other two data sources. And we'll be fine tuning a pre-trained universal sentence encoder model.


Spotify's Podcast Search Explained

So we use all that. And in the end, what we'll have is this podcast universal sentence encoder model, which we can then use to encode both our queries and our episodes.


Spotify's Podcast Search Explained

So there's one other. So we have our podcast episodes over here, place them into this transform. We've created those podcast episode descriptions.


Spotify's Podcast Search Explained

This other data source that you see over here is a manually created data source. Spotify did this and they use it purely for evaluating the model.


Spotify's Podcast Search Explained

They didn't use this in training. So they just manually curated a set of queries for particular podcast episodes.


Spotify's Podcast Search Explained

We will do the same. We'll just do, I think it's like seven queries and see how that works and use that to actually evaluate our model.


Spotify's Podcast Search Explained

Now, once we've done all of that, we come to the evaluation and also the actual, you know, how you use this.


Spotify's Podcast Search Explained

You would take your podcast model, you bring it over here into the middle here, and then you're going to take the evaluation data from your query generated data,


Spotify's Podcast Search Explained

bring it into here. You're going to encode everything and place it into a vector search engine. In this case, we're using Pinecone.


Spotify's Podcast Search Explained

This will allow us to store all of our episodes as embeddings, and then we'll be able to query and see if we are returning the correct episode based on our synthetic queries to episode pairs.


Spotify's Podcast Search Explained

And we'll also do the same using these manually curated query episode pairs. And then this is exactly what you would do in production as well.


Spotify's Podcast Search Explained

So after you've evaluated, you take all of your episode embeddings, put them in your vector database, so the Pinecone at the bottom there,


Spotify's Podcast Search Explained

and then you would let your users query. Your query would go into your podcast sentence transform model,


Spotify's Podcast Search Explained

and then it would be passed into your vector database, which would identify the most semantically similar, almost relevant episode vectors for you.


Spotify's Podcast Search Explained

OK, so that's very high level. That's how it all works. If it's confusing, no problem, because we're going to work through all of this.


Spotify's Podcast Search Explained

So now we'll move on to the actual implementation of all of this. Now, what we first want is a data set that is going to, as closely as possible,


Spotify's Podcast Search Explained

replicate the episode data that Spotify are using. So we need Kaggle for this. So you can download or install the Kaggle API using pip install kaggle.


Spotify's Podcast Search Explained

And then you'll need an account. So you need to sign up on kaggle.com and you can get an API key. So you just come over here, you click on top right,


Spotify's Podcast Search Explained

you'll see your profile. You go to, I think, account and you can come down and you have API here. So you'll be able to get like create new API token


Spotify's Podcast Search Explained

and it will download a kaggle.json file for you. And then what you do with that is if you try to import Kaggle, so you run this import kaggle,


Spotify's Podcast Search Explained

it will say, OK, you are not authenticated because you don't have kaggle.json in this directory. So all you do is put your kaggle.json in the directory that is specified.


Spotify's Podcast Search Explained

Once you've done that, you should be able to import Kaggle without any errors. And we'll move on to the data download. So we just authenticate our API.


Spotify's Podcast Search Explained

I'm not going to go too into depth here. There will be a link to this notebook so you can read through everything as you want.


Spotify's Podcast Search Explained

And all we're going to do is download these two datasets. OK, so if I let me copy this and I'll show you on kaggle this dataset.


Spotify's Podcast Search Explained

OK, so this is all podcast episodes published in December 2017 from this company called Listen Notes, which is a podcast search engine.


Spotify's Podcast Search Explained

So we download these podcasts and episodes data and we extract them because they'll be in zip files.


Spotify's Podcast Search Explained

And we can see, OK, this podcast is actually the details of the podcast show itself. And episodes is the details of individual episodes.


Spotify's Podcast Search Explained

And what Spotify did was concatenate the title and description from both of those and use that to create a sort of a single episode description that we then encode.


Spotify's Podcast Search Explained

So we do the same. We first need to merge those two data frames. So we're doing that based on the unique podcast ID.


Spotify's Podcast Search Explained

And we get this sort of data frame with everything we need inside it.


Spotify's Podcast Search Explained

I'm just doing a little bit data cleaning here, so stripping any white space from the features that we care about.


Spotify's Podcast Search Explained

So the title EP, description EP, title, podcast and description podcast.


Spotify's Podcast Search Explained

EP is just episode. And then I'm also just removing where we have any new values in any of those columns or features, just removing them because we have a lot here anyway.


Spotify's Podcast Search Explained

We don't even use all of these. So it's not a problem. And then I'm concatenating everything.


Spotify's Podcast Search Explained

I'm just putting a full stop in between each one of them and yeah, and then converting that into a list. So we just have like a list of these episode texts now.


Spotify's Podcast Search Explained

You see they do vary a lot. It's not the cleanest data, but it's usable.


Spotify's Podcast Search Explained

We also shuffle here as well.


Spotify's Podcast Search Explained

When you're shuffling, just bear in mind your shuffle is going to randomize the order. So later on when we have these curated queries, they won't necessarily be in the test set for you as well.


Spotify's Podcast Search Explained

So you may need to just make up some new queries.


Spotify's Podcast Search Explained

And at that point, we have our episodes data and we need to move on to query generation because we need query episode pairs.


Spotify's Podcast Search Explained

So Spotify used a fine or they fine tuned a BART model on MS Marco.


Spotify's Podcast Search Explained

Now I'm not going to fine tune the model because there are honestly so many query generation models that have been fine tuned on MS Marco already, including BART models.


Spotify's Podcast Search Explained

So I don't think there's really any point in fine tuning it because we're just replicating what we can just pull from Huginface hub.


Spotify's Podcast Search Explained

I tested a few different BART and T5 query generation models and I found this one to be the best for this particular use case.


Spotify's Podcast Search Explained

The queries were just more consistently like sensical, like they actually made sense.


Spotify's Podcast Search Explained

And it also supports multi or it has some multilingual support.


Spotify's Podcast Search Explained

I'm not sure how much, but it does manage to actually produce queries that make sense in different languages.


Spotify's Podcast Search Explained

So that's definitely pretty useful and aligns pretty nicely with what Spotify wanted as well.


Spotify's Podcast Search Explained

And then what I'm doing is so I'm using the transformers libraries, Huginface transformers.


Spotify's Podcast Search Explained

You can pip install transformers if you need to.


Spotify's Podcast Search Explained

I'm just initializing the tokenizer and the model.


Spotify's Podcast Search Explained

OK, so this is going to handle our query generation.


Spotify's Podcast Search Explained

On the end here, you see I put CUDA because I have a CUDA enabled GPU, which will make things a lot faster when we're doing this, because this can take a long time.


Spotify's Podcast Search Explained

And that's why I'm just taking the first 100000 episodes here.


Spotify's Podcast Search Explained

So we move on to the query generation loop.


Spotify's Podcast Search Explained

It's fairly long, sorry. This isn't like the quickest way you can do it.


Spotify's Podcast Search Explained

So a larger batch size means faster processing, but this is limited by the size of your GPU.


Spotify's Podcast Search Explained

And then number of queries I'm generating for each each episode.


Spotify's Podcast Search Explained

So. Spotify didn't specify what they use for number of queries here,


Spotify's Podcast Search Explained

but I'm using three because that's the in line with the approach taken by these other query generation techniques, GenQ and GPL.


Spotify's Podcast Search Explained

So I'm going to stick with that. Then we're just going through encoding everything in batches.


Spotify's Podcast Search Explained

So we're just tokenizing everything and then we're generating three queries per episode.


Spotify's Podcast Search Explained

Decoding those queries back to human readable text because this just outputs a load of token IDs and numbers.


Spotify's Podcast Search Explained

Decoding back to text and looping through and putting all those together.


Spotify's Podcast Search Explained

So the query and episode pairs and placing those together. Synthetic queries and episode pairs.


Spotify's Podcast Search Explained

And then we put all those in this pairs list.


Spotify's Podcast Search Explained

OK, and we can see a few here. So we have like a query and then we have an episode.


Spotify's Podcast Search Explained

No query, episode and so on. You can also see so here we have the multilingual support as well, which is very cool.


Spotify's Podcast Search Explained

And also again, yeah. So we now have a data source so we can use a fine tuning the model.


Spotify's Podcast Search Explained

So what if I talk about this in their article, which is here and they mention that they tried.


Spotify's Podcast Search Explained

OK, here. So they tried, but as I said, they decided they didn't really like it.


Spotify's Podcast Search Explained

So, you know, yeah, we're not using it for these multiple reasons I mentioned before.


Spotify's Podcast Search Explained

So what they did is use this universal sentence encoder model.


Spotify's Podcast Search Explained

And we're going to do something similar. But what I will note here is that they use TensorFlow Hub.


Spotify's Podcast Search Explained

And we can use TensorFlow Hub, but it means we can be stuck with TensorFlow and it makes our lives harder


Spotify's Podcast Search Explained

rather than using Hugging Face and PyTorch and the Sentence Transformers library.


Spotify's Podcast Search Explained

So rather than using the universal sentence encoder model, which we can't get from the Sentence Transformers library,


Spotify's Podcast Search Explained

as a pre-trained model, we are going to use a distilled universal sentence encoder model,


Spotify's Podcast Search Explained

which is literally like a smaller version of USC. So we'll come down here and we're using this one here.


Spotify's Podcast Search Explained

So the still USC based multilingual case. OK, so it's good.


Spotify's Podcast Search Explained

It's multilingual and it's still pretty much the same model as what Spotify used.


Spotify's Podcast Search Explained

So we just have the model details here. So the maximum number of tokens that will accept is 128.


Spotify's Podcast Search Explained

And it will output a 768 dimensional dense vector. So that's the meaningful vector that we spoke about earlier.


Spotify's Podcast Search Explained

Now, when we're fine tuning with the Sentence Transformers library, and by the way, if you do need to install that,


Spotify's Podcast Search Explained

you can just go pip install Sentence Transformers like this.


Spotify's Podcast Search Explained

Obviously in your terminal or using the exclamation mark at the start there.


Spotify's Podcast Search Explained

So when we're using Sentence Transformers library and we're fine tuning a model,


Spotify's Podcast Search Explained

we need to reformat our data, our input data, so our pairs, into a list of input example objects.


Spotify's Podcast Search Explained

Now, this input example object, the format for that varies depending on what task you're actually fine tuning with.


Spotify's Podcast Search Explained

Now, we're using a ranking task for fine tuning or optimising our model.


Spotify's Podcast Search Explained

And that means that we only need the query and the episode.


Spotify's Podcast Search Explained

OK, so all we do in that case, here as well, I'm just splitting so we have an evaluation set down here


Spotify's Podcast Search Explained

and a test set as well for later on when we're evaluating things.


Spotify's Podcast Search Explained

So all we need in our input example, which we have imported from Sentence Transformers, is the query and episode.


Spotify's Podcast Search Explained

Literally, so it's just a list of these input examples with query and episode in there.


Spotify's Podcast Search Explained

And you see the sample size that we have there.


Spotify's Podcast Search Explained

So let's continue.


Spotify's Podcast Search Explained

And yeah, so as I mentioned, we're going to be using a ranking optimisation function.


Spotify's Podcast Search Explained

Now, what does that look like?


Spotify's Podcast Search Explained

Given a specific query, like the one we used earlier, so eat better during Christmas holidays or Xmas holidays,


Spotify's Podcast Search Explained

our model is going to be given a set of episodes and it's going to need to rank the true episode


Spotify's Podcast Search Explained

for that particular query as number one in terms of scores.


Spotify's Podcast Search Explained

OK, so the true sort of pair for this query is this one here.


Spotify's Podcast Search Explained

And it will have to be able to identify that all these other episodes are not real pairs


Spotify's Podcast Search Explained

or are less semantically similar to this particular query.


Spotify's Podcast Search Explained

So the model learns to give higher scores to more semantically similar episodes for a particular query


Spotify's Podcast Search Explained

and lower scores to the more dissimilar episodes for a particular query.


Spotify's Podcast Search Explained

And that's why we don't need like a label in this case.


Spotify's Podcast Search Explained

We don't actually need a particular similarity label or anything when training our model,


Spotify's Podcast Search Explained

which is a really nice sort of benefit of using ranking functions.


Spotify's Podcast Search Explained

You literally just need pairs of data and you can train with that, which is pretty cool.


Spotify's Podcast Search Explained

Now, the way that model actually needs to compare the queries and episodes is by taking a query,


Spotify's Podcast Search Explained

converting it into a query vector, taking the episode, converting that into an episode vector


Spotify's Podcast Search Explained

and placing them in a vector space and comparing the similarity between them.


Spotify's Podcast Search Explained

Now, the way that we are going to be calculating similarity in this case is using cosine similarity.


Spotify's Podcast Search Explained

So we're essentially calculating the similarity or the angle between vectors.


Spotify's Podcast Search Explained

So this is this in the middle here. That is our query vector.


Spotify's Podcast Search Explained

And this little cone is representing the angle in a 3D space around that query vector.


Spotify's Podcast Search Explained

These two episode vectors here, episode embeddings are the two most similar.


Spotify's Podcast Search Explained

So we've got top k equals two over here. That's because we're trying to return the two most similar.


Spotify's Podcast Search Explained

And these other vectors, like this one, for example, in terms of actual distance,


Spotify's Podcast Search Explained

this one might actually be closer than this one. But in terms of angular distance, it's further away.


Spotify's Podcast Search Explained

So that's why we have these two embeddings being selected rather than any of these other ones.


Spotify's Podcast Search Explained

So the model needs to create these vectors and it needs to learn how to put similar meaning


Spotify's Podcast Search Explained

query and episodes in a similar vector space and dissimilar query and episodes in as far apart as possible.


Spotify's Podcast Search Explained

Now, another thing that we need to consider here is because we're using a ranking function,


Spotify's Podcast Search Explained

which is going to work by taking a query and then a batch of episodes,


Spotify's Podcast Search Explained

we need to make sure that within that batch of episodes, we don't have any duplicates.


Spotify's Podcast Search Explained

Because imagine, so we're going to optimize this model and we've got a query and episode pair.


Spotify's Podcast Search Explained

And we're saying, OK, for this query, this episode is the most similar.


Spotify's Podcast Search Explained

We're going to optimizing the model based on that.


Spotify's Podcast Search Explained

If we have the exact same episode further down in that list and we're telling the model,


Spotify's Podcast Search Explained

actually, OK, these two episodes are exactly the same, but one is right and one is not.


Spotify's Podcast Search Explained

Our model is just going to be confusing. That doesn't make sense. It can't do that.


Spotify's Podcast Search Explained

So we have to make sure that we don't have duplicates in any of the training batches.


Spotify's Podcast Search Explained

Now, it's probably not a problem if you have the odd one, but we don't want any.


Spotify's Podcast Search Explained

We just want to be certain there's nothing in there.


Spotify's Podcast Search Explained

So we can actually do that using Sentence Transformers No Duplicates Data Loader,


Spotify's Podcast Search Explained

which will handle removing any duplicates from batches for us.


Spotify's Podcast Search Explained

As well, we use a batch size of 64. So if we consider the training optimization function here,


Spotify's Podcast Search Explained

we're taking query and then we're taking a batch of episodes.


Spotify's Podcast Search Explained

Imagine we have a batch size of three, right? Or even two. We have a batch size of two.


Spotify's Podcast Search Explained

We're just comparing our one query against two episodes.


Spotify's Podcast Search Explained

Our model can get, even if it's just randomly guessing, it can get the right answer around 50% of the time.


Spotify's Podcast Search Explained

With such a small batch size. If we increase that to 100,


Spotify's Podcast Search Explained

then the model is going to perform a lot more poorly, randomly guessing.


Spotify's Podcast Search Explained

So the consequence of that is that using a larger batch size for this particular training method


Spotify's Podcast Search Explained

is going to typically result in you having a better performing model


Spotify's Podcast Search Explained

because it makes the task of ranking harder for the model.


Spotify's Podcast Search Explained

So your model must get better in order to actually accurately rank everything.


Spotify's Podcast Search Explained

So we increase the batch size as much as our hardware will allow us to.


Spotify's Podcast Search Explained

Now, after that, we can initialize the loss function.


Spotify's Podcast Search Explained

This is where the ranking optimization comes in.


Spotify's Podcast Search Explained

So in Sentence Transformers, the ranking function is multiple negatives ranking loss.


Spotify's Podcast Search Explained

Okay, so we initialize that.


Spotify's Podcast Search Explained

There is just one more step before we actually move on to fine tuning.


Spotify's Podcast Search Explained

So if we have a look over here and we go down to offline evaluation.


Spotify's Podcast Search Explained

In order to evaluate the model, they use two different types of metrics, in batch metrics and full retrieval.


Spotify's Podcast Search Explained

We're going to do both. So for the in batch metrics,


Spotify's Podcast Search Explained

so just calculating recall and MMR at the batch level.


Spotify's Podcast Search Explained

Okay, so what we can do to replicate that is use this re-ranking evaluator from Sentence Transformers.


Spotify's Podcast Search Explained

And we use that to perform ranking and calculate the MMR metric using our evaluation set.


Spotify's Podcast Search Explained

So just one thing, as we have already done here is we've removed duplicates from batches.


Spotify's Podcast Search Explained

We need to do the same before we actually feed data into our re-ranking evaluator.


Spotify's Podcast Search Explained

Remember there are a lot because we create three queries per episode.


Spotify's Podcast Search Explained

So we definitely do need to remove those.


Spotify's Podcast Search Explained

So remove any duplicates using this.


Spotify's Podcast Search Explained

And in the end, we have a thousand unique pairs for our evaluation set.


Spotify's Podcast Search Explained

So then we feed them into the re-ranking evaluator, which requires a particular format of query.


Spotify's Podcast Search Explained

Any positives we have. So we just have one positive per query here.


Spotify's Podcast Search Explained

It's just a single list and a negative.


Spotify's Podcast Search Explained

And then in here, we just need to pass in all of the other episodes that are not the positive episode for that particular query.


Spotify's Podcast Search Explained

So we do that in here.


Spotify's Podcast Search Explained

And then we initialize a re-ranking evaluator.


Spotify's Podcast Search Explained

And I set MMR at K to consider the top five items when it's calculating its score.


Spotify's Podcast Search Explained

I'm not going to go into the metrics because it takes a little bit more time.


Spotify's Podcast Search Explained

So calculating the performance.


Spotify's Podcast Search Explained

So this is the MMR at five for the model without any fine tuning.


Spotify's Podcast Search Explained

So we get 0.68.


Spotify's Podcast Search Explained

OK, so at this point, we're actually ready to go ahead and fine tune our model.


Spotify's Podcast Search Explained

And this is basically the easiest set.


Spotify's Podcast Search Explained

So we set the number of epochs to one. I did try other, so like more epochs.


Spotify's Podcast Search Explained

Typically with sentence transformers, you actually only really need to train for one epoch depending on the model and the data.


Spotify's Podcast Search Explained

Anything more, it actually degraded the performance.


Spotify's Podcast Search Explained

So we went one epoch here.


Spotify's Podcast Search Explained

And we always or typically use a number of warm up sets.


Spotify's Podcast Search Explained

So here for the first 10 percent of training sets, the learning rate is going to be slowly increasing up to the default learning rate.


Spotify's Podcast Search Explained

Which I think is like one E to the minus five or something along those lines.


Spotify's Podcast Search Explained

And then we're just saving the model that we train into this directory here.


Spotify's Podcast Search Explained

So this WSE podcast NQ.


Spotify's Podcast Search Explained

NQ here is natural query.


Spotify's Podcast Search Explained

And then, yeah, so that trains.


Spotify's Podcast Search Explained

It shouldn't take too long. I think maybe this took an hour, maybe two hours to train.


Spotify's Podcast Search Explained

So it's pretty quick.


Spotify's Podcast Search Explained

And with that, we can actually look at the evaluation.


Spotify's Podcast Search Explained

So how did this model perform on the evaluator that we initialized at the start?


Spotify's Podcast Search Explained

And we can actually see that. So I'm in another environment here where I actually trained the model.


Spotify's Podcast Search Explained

We go to this still usc podcast NQ folder.


Spotify's Podcast Search Explained

And then he will see this eval directory.


Spotify's Podcast Search Explained

And here open this and we can see we have.


Spotify's Podcast Search Explained

Over on the right here, the MMR at five value and it's 0.888.


Spotify's Podcast Search Explained

So that is a big improvement from what we had up here, which is 0.68.


Spotify's Podcast Search Explained

So it's a 20 point increase, which is really good.


Spotify's Podcast Search Explained

That's on small batches. So that doesn't really replicate the real world use of this model.


Spotify's Podcast Search Explained

So what we want to do in the final evaluation step is replicate that.


Spotify's Podcast Search Explained

So what we're going to do is take the test data we had, which has many thousands of episodes in there.


Spotify's Podcast Search Explained

We're going to encode all of those.


Spotify's Podcast Search Explained

And then we're going to perform a semantic search as we would expect this to be used in real life.


Spotify's Podcast Search Explained

So what we'll do is set up the vector database with thousands of episode vectors in there from our test data set.


Spotify's Podcast Search Explained

And then we're going to calculate the recall value this time for our fine tuned model and also our not fine tuned model.


Spotify's Podcast Search Explained

So we initialize pine cone, create a evaluation index here. Make sure we use the cosine metric.


Spotify's Podcast Search Explained

We connect to that. Oh, if you if you need an API key, you need to go to the pine cone.


Spotify's Podcast Search Explained

The IO is free. So this is all free. There's nothing you don't need to worry about anything there.


Spotify's Podcast Search Explained

We create our index, we connect to it here.


Spotify's Podcast Search Explained

And then what we can do is so before we index our test data, we remove duplicates like we did with the evaluation set earlier.


Spotify's Podcast Search Explained

Same thing I'm not going to go through again. And then here. So what I'm doing is going through again,


Spotify's Podcast Search Explained

just make sure there's definitely no duplicates in there. There shouldn't be any way.


Spotify's Podcast Search Explained

But just in case going through and creating batches of episodes to actually encode here using our new model.


Spotify's Podcast Search Explained

This is using the Distill USC podcast NQ model. And then we insert them into our pine cone vector database.


Spotify's Podcast Search Explained

And then just refreshing the batch and doing it again all in batches.


Spotify's Podcast Search Explained

And then at the end, I'm just going to look at the index stats so we can see how many vectors we actually have in there, which is 18 and a half thousand.


Spotify's Podcast Search Explained

Then what I'm doing here is I'm going to calculate the recall at K.


Spotify's Podcast Search Explained

So I'm going to loop through all the queries we have. So all the queries from our test set.


Spotify's Podcast Search Explained

So these are synthetic queries first. And just looking at, OK, what is the recall using those synthetic queries?


Spotify's Podcast Search Explained

Now it's 0.88, which is like incredibly good, but we're using synthetic queries here.


Spotify's Podcast Search Explained

So it's not really representative of real human queries. So this is where the curated queries come in.


Spotify's Podcast Search Explained

So I went through, I found different episode descriptions, which were in the indexes 1, 8, 14 and so on.


Spotify's Podcast Search Explained

And I just created like a query for each one of those episodes. That makes sense to me.


Spotify's Podcast Search Explained

OK, now this isn't perfectly accurate. We could use this and we could return another episode that's actually more relevant.


Spotify's Podcast Search Explained

But using what I've done here, it would show up as not a match and bring down the score.


Spotify's Podcast Search Explained

So just bear that in mind as well. So once we use those curated, like true query episode pairs or more sort of human queries,


Spotify's Podcast Search Explained

we get a recall at K of 57. Now, it's definitely not as impressive as the 88 we had before. It's OK, but it's nothing special.


Spotify's Podcast Search Explained

But what I want to do here is just compare this to what we had before. So the pre-trained model without any fine tuning.


Spotify's Podcast Search Explained

OK, so to do that, I'm just creating a new index. If you're using the free version of Pinecone, you can only have one index at a time.


Spotify's Podcast Search Explained

So you will have to you can you can delete an index, you know, so you can then refresh it and do this.


Spotify's Podcast Search Explained

So what we do is I'm going to create the I'm going to initialize the pre-trained model that hasn't been fine tuned on our particular data set.


Spotify's Podcast Search Explained

And here I am creating a new index, eval 0, connecting to that index.


Spotify's Podcast Search Explained

And I'm going through doing the exact same thing as I did before. OK, so just pushing all of those vectors or episode embeddings that have been encoded using the older model that hasn't been fine tuned.


Spotify's Podcast Search Explained

And then calculating the recall at K again using that, using the curated data set. And we get a terrible score of 0.28.


Spotify's Podcast Search Explained

So we can see there's actually a massive improvement from 28 to 57. That's a 29 point improvement, which is huge, especially to say that our training data was synthetic.


Spotify's Podcast Search Explained

Right. So a synthetic data set is never going to perform as well as a genuine data set.


Spotify's Podcast Search Explained

So if we had those other two data sources that Spotify use, we could probably create something that is like very impressive compared to what we've done here.


Spotify's Podcast Search Explained

But to say we just use episode data without any queries and synthetically create queries, I think this is really impressive.


Spotify's Podcast Search Explained

And it shows that Spotify's methodology, at least in terms of adding these synthetic queries, definitely does contribute to the overall model.


Spotify's Podcast Search Explained

And it shows us a really cool way to, without any real training data, do this.


Spotify's Podcast Search Explained

Now, another thing that we should really think about here is how well does this scale?


Spotify's Podcast Search Explained

So we're only using 18 and a half thousand vectors in this evaluation case.


Spotify's Podcast Search Explained

Using Pinecone, we're quite easily able to go up to hundreds of millions of episodes or vectors in there and even billions.


Spotify's Podcast Search Explained

So in terms of scalability, that is, well, unless you're going into this sort of trillion scale, this is very possible with pretty much every data set that you're going to be using.


Spotify's Podcast Search Explained

And it will still be incredibly fast, which is, I think, really cool.


Spotify's Podcast Search Explained

I think, you know, that's really awesome.


Spotify's Podcast Search Explained

The fact that we can do this, no data, make it incredibly scalable and build something really, really awesome.


Spotify's Podcast Search Explained

So that's it for this sort of walkthrough.


Spotify's Podcast Search Explained

I will say just if you are going to build something like this with Pinecone and doing all this sort of cool vector search stuff,


Spotify's Podcast Search Explained

let me know because over at Pinecone, we are looking for people that would like to showcase the work that they're doing.


Spotify's Podcast Search Explained

And it would be really cool to see what you're building.


Spotify's Podcast Search Explained

And if you'd sort of like to share that and get other people seeing what you're building, I think that's a really good way to do that.


Spotify's Podcast Search Explained

So if you're interested in that, just go over to the community page at Pinecone and you'll be able to find how to submit any projects that you're working on there.


Spotify's Podcast Search Explained

So I hope all this has been useful and interesting.


Spotify's Podcast Search Explained


Ultra-efficient Classifier Fine-tuning with Vector Search

Today we're going to talk about a more effective way of training classification models.


Nowadays pre-trained models dominate the field of machine learning.


There are very few ML projects that start with us actually training a model from scratch.


Instead we usually start by looking for a off-the-shelf pre-trained model.


Whether that pre-trained model is from a online platform like Pytorch hub or Hugging Face hub,


or from our own internal already trained in-house models.


The ecosystem of these pre-trained models whether external or internal


has allowed us to push the limits of what is possible in machine learning.


This doesn't mean however that everything is super easy and everything works all the time.


There are always going to be some challenges.


Fortunately we're able to tackle a lot of these problems that are actually shared across


a huge number of pre-trained models because they tend to have similar points of failure.


One of those is the excessive compute and data needed to actually fine-tune one of these models.


Now focusing on classification a very typical scenario that we have is that we have some model,


some big model like BERT or T5 and what we want to do is fine-tune this model for classification.


Now one way we can do that is we add a simple linear layer onto the end of it


and then we fine-tune that linear layer.


Now what I want us to focus on here is the model that comes before doesn't really matter.


We only really care about this linear layer.


We can actually fine-tune that for a lot of different use cases


without even touching the model weights of the big pre-trained model that comes before it.


It's the classification layer that is actually producing the final prediction.


And because of this that classification layer can become the single point of failure


in producing our predictions.


So we focus on fine-tuning that classification layer and a common approach to doing this


might look a little bit like this.


First we have to collect a data set that focuses on enabling this model to adapt to a new domain


or just dealing with data drift.


Then we have to slog through the data set and if it's going to work well it's usually a large data set


labeling the records as per their classification.


And then once all records have been labeled we have to fine-tune the classifier.


This approach works but it really is not efficient.


There's actually a much better way of doing this.


What we need to do is focus our fine-tuning efforts on the essential records that actually matter.


Otherwise we're wasting time, our own time and compute on annotating and fine-tuning


across the entire data set when the vast majority of the data in the data set probably doesn't matter.


So now the question is how do we decide which samples are actually essential and which are not?


Well that's where we can use vector search.


We can use vector search to search through our data set before we even annotate anything


and identify the records that are going to make the biggest impact on our model performance.


Meaning we save our time and a lot of compute by just skipping the non-essential records.


Some of you may be thinking what does vector search have to do with training a classification model?


Well it's actually super important.


Many state-of-the-art models are available as pre-trained models.


Those are models like BERT, T5, Epsilonet, OpeningEyesClip.


And these models use an insane number of parameters and perform a lot of complex


operations. Yet when applied to classification, we're actually relying on the final layers


that are added onto the end of these huge models.


Okay so we might have some simple feed-forward layers or just a linear classification layer.


Now the reason for this is that these models, they're not being trained to produce class predictions.


We can think of them as actually being trained to make vector embeddings.


So we pre-train these big models and the idea is that after pre-training these models will produce


these very information-rich vector embeddings.


And then what we do for different tasks is that we add an extra task-specific head onto the end of that.


And that task-specific head is taking that vector embedding or vector embeddings from the model


and running them through a smaller network.


You know like I said it can just be a linear layer and outputting something else.


Okay outputting those predictions.


So the power of these models is not that they can you know they can do classification,


question answering, all these different things.


The power of these models is that they produce these very information-rich vectors


that then smaller simpler models can use to do these tasks of question answering, classification and so on.


These vectors that these models are producing are simply full of useful information


that have been encoded into a vector space.


Okay so you can imagine it in this vector space, imagine a 2D space.


We have a vector A here, vector B here.


Those two are very close to each other and therefore they share some sort of similar meaning.


Whereas vector C over here is very far away from from A and B.


Therefore it shares less meaning with A and B.


Now the result of this is that these models are essentially creating a map of information.


Using this map they're able to consume data like images or tech and output these useful


information-rich representations with vectors.


So our task in classification now is not to consume data and try and abstract different meaning from that


and classify that abstraction of meaning.


In reality the abstraction of meaning is already handled by the big models.


Instead our task with classification is to teach a smaller model to identify the different regions


within that map or the vector space.


Now a typical architecture that we will see for classification is a pre-trained model followed by a linear layer.


Now we can think of the internal weights of this classifier as actually being a vector within the wider vector space.


And Ido Liberty, the founder and CEO of Pinecone and past head of Amazon AI Labs,


explained to me that we can actually use this fact and couple it with vector search in order to


massively optimize the learning process for our classifier.


So what we need to do is really imagine this problem as being within a vector space or a map.


We have the internal model weights W and we have all these vectors that as of yet are unannotated


and we haven't fine-tuned on them yet.


We want to calculate the dot product between W and X.


If they share a positive direction they will have a positive value


and they produce a negative score if the directions are opposite.


Now there is just one problem with dot product here.


It considers both direction and magnitude which means that if we have a vector X that has a larger magnitude


than another vector X even if that other vector is actually the same vector as our model weights or very similar,


it can actually output a larger dot product score.


So what we need to do is normalize all these vectors that we're comparing.


This simply removes the magnitude problem and makes it that we are comparing only the direction of the vectors.


Now when we fine-tune the linear classifier with these vectors it's going to learn to align itself with vectors


that we label as positives and move away from vectors we label as negatives.


Now this will work really well but there are still some improvements that we could add in here.


First imagine we return only irrelevant samples in a single training batch.


They will all be marked as negative one and the classifier knows to move away from these values


but it doesn't know in which direction.


Okay and especially in a high dimensional space there are a lot of directions that the classifier can move in.


So this is problematic because it means that the classifier is just going to be moving at random away from those negative vectors.


Another problem is that many labels be more or less relevant.


So imagine we had the query dogs in the snow and then we had two pieces of text a dog and a dog in the snow.


Both of those are relevant depending on what you're looking at but a dog in the snow is more relevant.


These two pieces of text are not equally relevant but at the moment all we can do is label one as negative one as positive or both as positives.


And that's not really ideal because it doesn't really show the full picture of both of these are relevant just one is more than the other.


So what we need is almost like a gradient of relevance.


We need a continuous range from negative e.g. minus one to positive e.g. plus one.


Even if we just have a range from negative one to negative 0.8 there's still a direction that the model can figure out from that range of values.


So all of this together just allows our linear classifier to learn where to place itself within the vector space produced by the model layers preceding it.


Now that describes a fine tuning process but we can't do this across our entire data set.


If we have a big data set which you probably do it would take too much time annotating everything and it would be a waste of our time as well.


To do this efficiently what we must do is capitalize on the idea of identifying relevant versus irrelevant vectors within a proximity of the model's learned weights w.


So we focus our efforts on this specific area that is actually going to be helpful.


For an already trained classifier those are going to be the false positives and false negatives predicted by the classifier.


However we also usually don't have a list of false negatives and false positives.


But we do know that the solvable errors will be present near the classifier's decision boundary.


E.g. the line that separates the positive predictions from negative predictions.


So we use vector search in order to actually pull in the high proximity samples that are most similar to the model weights w.


We then label those vectors and use them for training our model.


The model optimizes those internal weights w.


We extract them again and then we perform a vector search with them again.


And we just keep repeating this process over and over again until the linear classifier has been optimized and is producing the correct predictions that we need.


So by focusing annotation and training on these essential samples we avoid wasting time and compute on those vectors that don't make as much of a difference.


OK so all of that is the general idea behind this process.


Now let's have a look at how we can put all that together and fine tune a classifier with vector search.


Now we will see that there are two parts to the training process.


First we need to index our data.


So that is where we embed everything using the preceding model layers e.g.


BERT or CLIP or so on and then saw those in a vector database.


And then step two is that we actually fine tune classifier.


So query with model weights w, return the most similar records, annotate them and then use them to fine tune the classifier.


So let's go ahead and start with indexing.


Given a data set of images or other formats we first need to process everything through the big model proceeding our linear classifier to create the vector embeddings.


For our example we're going to use a model called CLIP that is capable of understanding both text and images.


And it has been trained on text image pairs and has learned how to encode them into as similar vector space as possible.


So what we're going to start with before indexing anything is initializing a data set that we can then encode with CLIP.


So we're going to use this data set from Hugging Face data set hub so we can pip install everything we're going to need for this here.


We're taking the train split and that contains 9.5 thousand images.


Some of those are radios like you can see here.


There's pictures of dogs, troops and a few other things.


And we can see an array of one of those images right there.


Now it's not so important for what we're doing here.


What we do want to do is actually initialize both the model and the pre-processing steps before the data is being fed into the model.


So we do that here. So initialize the model CLIP using this model ID here.


OK so this is one version the CLIP model.


And then the pre-processor will just take images and process them so that CLIP can read them.


OK that's all we're doing here. We're going to go through all of these steps.


This is the pre-processing and from that we get the image features.


OK those image features are a vector representation of the image.


OK so in this case we've done the Sony radio image and that gives us a 512 dimensional vector embedding.


OK. The embeddings from CLIP are not normalized.


OK so we're going to be using dot product both within the model and during our vector search.


So we should really normalize these. So we do that here.


And then we see that these values are all between the values of negative one to two plus one.


Now that's how we embed or create a vector embedding for a single item.


But we're going to want to do loads of items and we're also going to want to index them and store them inside a vector database.


So we're going to use Pinecone for this. You may need to sign up for a free API key if you haven't already.


And what we do is initialize our connection to Pinecone here. You just put your API key here. It's all free.


And then we create an index. Now it's important that we have a few things here.


So the index name that doesn't actually matter. OK. You can put whatever you want.


But what you do need is the correct dimensionality. So that is the 512 that you saw up here.


That is what we put in here. We do need to make sure that we're using dot product similarity.


And we're going to also include this metadata config. So basically when once we see an image and we label it.


We're going to tell Pinecone we don't want to return that image again.


So that we can go through and not over optimize on like 10 images.


And then we connect to the index after we have created it there.


Now to add that single feature embedding that we just created, that image embedding we just created.


We would do this. OK. So we have an ID and then we just convert the embedding into a list format.


And we just upset. So with that we have one embedding within our vector index.


But of course we want to have our full data set in there so we can search for it and add data and so on.


So to do that we're going to use this loop here. I'm not going to go through it because it's literally what we've just done.


OK. The only thing I think about it here is this which is checking for grayscale versus RGB images.


But the rest of this is exactly the same. OK. We're just going to assume it all at a larger scale.


And we're also adding in the metadata here. OK. So that scene we're setting it to zero for all the images to start with.


And then we'll set it to one once we've seen a set of images mark them as you know positive or negative and train with them.


Then we set that scene value to one so we don't return it again. OK. So we have this this radio.


Let's have a quick look at how we might query. So we create our query vector XQ here which is just we're doing the same thing again.


So we did before normalizing it and then we query with it. OK. And that returns these items here from Pinecone.


Let's have a look at what they look like. So the first one is obviously that radio. That radio is the most similar other vector.


So naturally that would be the first thing that gets returned. OK. Next one we have a car radio.


We have another Sony radio. I think it's even the same model and another Sony radio which is also the same model.


It seems so. And then just another radio. It's very similar. So clearly those embeddings are pretty good from clip.


But now all we want to do is fine tune a linear classifier on top of that to classify these different images.


OK. So to do that I'm going to start from scratch. So it's a new notebook. You can find all the links to these notebooks by the way in the video description.


Or if you're watching this on the article down at the bottom of the article in the resources section.


So here initialize the connection to the index again. You don't need to do this if you just ran through the last bit of code.


You can just keep that as it is and maintain your connection to the index. Again we're going to load the data set.


And again you don't need to do that if you've already done it. Initialize the model so clip and the processor.


So there's one thing different here and you can actually tokenize using the other pre processor.


But for the sake of covering everything I'm just showing how to do with the clip tokenizer fast here as well.


So here we're initializing just the tokenizer side of the clip preprocessor. And we're setting up this prompt.


So dogs in the snow. We tokenize them to get a set of token IDs and then we use the model get text features method in order to get a vector embedding of that text.


That dogs in the snow prompt. OK. We come down here. We create the query vector from that.


And we're going to retrieve top 10 most similar records and saw them in XC. So it's just like the contents.


So there's a few things in XC here. We actually don't need all of this.


So what we want is the IDs and then the values as well. So first we get the IDs and then get the values.


OK. And we can see what's returned. So dogs in the snow. This one is not a dog in the snow but you can kind of see where it's a bit confused.


The sand in the background does look kind of white and snowy. But then the rest of these.


Yeah they're dogs in the snow other than this one. So it's returning the right thing here.


But let's say we don't want dogs in the snow. OK. And I say we want to adjust this to something slightly different like for example dogs at dog shows.


And we'll go through this. So this code here. Not really that important.


All this is is a little interface I built within Jupiter so that we can sort of quickly go through and label the images.


So I would run this. I'm not going to run it again. So I'll just run this here.


And basically what you can do is it's going to show an image. So sample this one here.


It's going to show the image and say OK what you rate this from negative one to one.


And you just go through you say what you what you would rate it.


And then that will give you or that will basically produce a dictionary that maps these ideas to the score that you gave it.


So you can see all the scores I gave last time I run this. And you can just double check that the ideas and scores are aligned here.


Yes they are. So you don't need to worry so much about that.


And all we do is we need to get the values which are going to be the inputs of training data for the linear classifier.


And then we get the labels. OK so the scores. So we go through and what we're going to do here is just initialize a PyTorch linear classifier layer.


And what I do first is so in most cases I imagine that we're going to have a linear classifier already trained.


So I'm just emulating that here. So I'm getting the query vector reshaping that and I'm inserting it as the first set of model weights w.


And what we're going to do is we're going to initialize the loss. We're going to use bc with logits loss and we're going to use stochastic gradient descent.


Now this learning rate you'll probably find that's quite high and it is high.


We're just kind of putting it high so that we can see a lot of like quick movement through the data set. If you actually implementing something like this you might want to use a lower learning rate.


So with that we just create this function fit here which is basically just a training loop.


And we can set the number of iterations per training loop. Again you might want to lower this if you don't want to move so quickly through the vector space and keep things a bit more stable.


And yeah we just call fit. From that the model weight will actually be optimized and it will change.


And that will represent the next query that we're going to pass into our vector database.


So we convert into a flat list so that pinecone can we can query in pinecone with it.


And so that we're not returning the same records that we just went through we update the metadata attached to each one of the vectors that we've just seen to be set to equal scene equals one.


And then the reason we do that is because we add a filter now to the next query where we set scene equal to zero.


And then we return the next set of queries and see here we have some other images.


And basically what I'm doing here is trying to optimize for dogs in fields.


And then from dogs in fields we're going to try and move to dogs at dog shows.


And we'll just go through this bit quickly now. So this is just tuning some putting order what we just did into a single function just make things a bit simpler.


And yeah we'll go through. OK. So you can see how things are kind of changing more towards dogs and fields here.


And then here it goes a bit crazy because basically I'm putting a lot of dogs as negative.


So now it's thinking or maybe I don't actually want to see any dogs.


And that makes it push away from that. But obviously I don't want that to happen.


So I just set everything negative here other than I think this image that has a field or maybe this image that has a field and also this image of a dog.


And then we go towards dogs again. Focus on that push towards dogs.


And then here you can see the first in the middle right here. There's the first image of dogs at a dog show.


Actually I think this is also a dog show here. So that would technically be the first one.


But this is what I'm looking for. More like this sort of image.


So we focus on that. We push that a little more. Next one we see.


Oh OK. We have a few more dog shows here. So here and here.


And we keep pushing for that. And you can see as we go through each step there's more of these dogs in dog shows because that's what I'm labelling as being more relevant.


OK. And now we're really getting into that sort of space. Keep going.


And now we're at a point where pretty much everything when we're returning is a dog show.


So this is the final bit. So now that we've done that we want to set all of the scene labels in our vector database back to not seen.


OK. Because we want to search again. We can either search without a filter just to check that it has trained the classifier.


Or we just reset all of those scene labels. If you want to go through data again and focus more on those.


That's where you might want to reset all the labels back to zero. So to do that all I'm going to do is go through a while loop.


And we keep going through and we search for everything where the filter is equal to seen.


We get those IDs and then we mark them as not seen.


Once we don't return any more items that means we've set everything to not seen because we're not returning anything else we've seen equal to true.


So at that point we break. So after that if we search again we get a completely unfiltered view of the search results.


And here we go. OK. So we can see loads of dogs at dog shows.


Now there's one here that isn't a dog at a dog show. I think the rest of them are.


So with that we've actually fine tuned our classifier. So now that we've finished optimizing those model weights we can save them to file.


OK. So we do this. And with that let's have a look at how the model performs on actually classifying images.


So again move to another notebook. This is number 02 classifier tests. And here we're going to test the classifier on a set of images that it has not seen before.


So again we initialize everything again if you've already loaded everything and you're in the same notebook you don't need to do this.


So we need to load the validation split from ImageNet. So you can see here this before was train.


Now it's validation. So you all need to rerun this bit. And we have about 4000 images there.


Now let's start by checking the predictions for some specific images. OK. So this one is a dog at a dog show.


So we preprocess that. We get the image features from clip. And then we make a prediction.


So the classifier and then we put in those the vector output by clip. And you can see there's a pretty positive value there.


So positive remember is a true value. Negative is a not true prediction. OK. Cool.


So that's a that's correct. It's predicted that that is a dog show. Now let's have a look at this.


OK. This is not a dog show. So we should see that it will predict a negative value.


So let's go through and yeah we get a pretty negative value there. So we can labeled full data set and we'll find a


cut off point between what is viewed as is relevant and what is irrelevant. So basically anything is positive.


So we do that here. I'm not going to go through it but it's essentially the same thing as what we just said.


I'm just making a list of these predictions. OK. I'm going to add a column to the image at data set called predictions.


So we now have these these three. And let's have a look. So filter out any results where the prediction is not positive.


So we get 23 results. And let's have a look at what those are. So those 23 positive results.


All of them I think almost all of them are dog shows. And we keep going through.


So each one of these as we go through is it's been scored less highly but all these sub scored very highly.


OK. Going through. And then we go through and then yeah we get this I don't know emoji chainsaw thing which is right at the bottom of these


positively labelled things kind of random. I don't know why it's in there. Yeah. So other than the literally these two images right at the end


everything else is a true positive. So this predicts everything correctly other than these two.


This one no idea why. This one I kind of understand you know dogs in a field. So generally speaking these are I think very good results.


And we got these from fine tuning our classifier on not really that many images. I think there was maybe 50 images there.


So really good results on a very small amount of data. And that's because we use Invex research to focus our annotation and training on what is the most important part of the data set.


Now doing this for an image classifier is just one example. We can do this with text. We can do this in recommendation engines or even anomaly detection.


There's like a huge number of use cases with this basically whenever you need to classify something and you want to do that efficiently you can you can use this as long as using something like a linear classifier.


So for me I think that was a really cool method for efficiently training classification models.


Thank you a lot to Edo for actually sharing with us and explaining and walking me through everything.


I think yeah this is a really useful method and I hope you will find it useful as well.


Is GPL the Future of Sentence Transformers? | Generative Pseudo-Labeling Deep Dive

Today we're going to dive into what I think is probably one of the most


interesting techniques in semantic search for a very long time. That


technique is called generative pseudo labeling or GPL. It's from Noah's Rhymer's


team at the UKP lab and this is primarily the work of Ketsen Wang.


Ketsen Wang also produced the TSV paper maybe around a year or so ago as well.


TSV is the unsupervised training method for sentence transformers and


GPL kind of goes takes that and kind of puts up another notch. So we're going to


be using GPL to take unstructured data and generate structured data from it for


training sentence transformers. Now before we really dive deep into into GPL


I want to give a little bit background and set the scene of where we might want to


use this technique. So in 1999 there was a concept known as Semantic Web


described by the creator of the World Wide Web Tim Berners-Lee. Now Tim Berners-Lee


had this dream of the web in the way that we know it today but where you have


machines roaming the web and being able to just understand everything.


But when you start to become a bit more niche it gets hard to actually find that


sort of data set and pretty much impossible most of the time. So for


example let's say you're in the finance industry you have these internal


documents but they're fairly technical financial documents and you want to


build a question answering system where people or staff can ask a question in


natural language and return answers from those documents. Okay it's going to be


hard for you to find a model you do have financial Q&A models but they've been


fine-tuned on like personal finance questions on Reddit and okay kind of


similar but not really. Like personal finance and technical finance


documentation in a finance company very different. So it's very hard to actually


find a data set that is going to satisfy what you need in that scenario because


it's fairly niche and it's not even that niche there's a lot of companies that


need to do that sort of thing. Another example is a project I worked on was for


the Dewey language we wanted to create a set of language models for the Dewey


language which is the national language of the Maldives. Now there's not that


many Dewey speakers worldwide so it's pretty niche. So it's very hard to


find labeled data and I think what we ended up doing is actually using


unsupervised methods we use TSVE for the actual sentence transformer


model. What we have there is another use case where it's fairly niche and it's


hard to find labeled data for but there is unlabeled data, unstructured data in


both of those scenarios. In your finance company you have the


documents there you just don't have them they're not labeled in a way that you


can train a model with or in the Dewey language example we have there is many


web pages in Dewey that we can scrape data from but it's not labeled. So for


these more niche topics which cover the vast majority of use cases you really


either need to spend a lot of money labeling your data or we need to find


something else that allows us to either synthetically generate data or just


train on the unlabeled data. Now training on the unlabeled data doesn't really


work okay it works to an extent TSVE showed that it can but you're looking at


very generic semantic similarity and it's not perfect. GPL again it's


not perfect but it's definitely much better. So very you know back to the


semantic web example in the case of the semantic web the majority of the


semantic web is going to be these niche topics that we don't have labeled data


for. GPL is not a solution it's not going to create the semantic web or anything


like that but it does allow us to actually you know target those previously


inaccessible domains and actually begin producing models that can intelligently


comprehend the meaning behind the language in those particular domains


which I think is super fascinating and the fact that you can actually do this


is so cool. Now there is a lot to it this video I'm sure will be quite long


but we really are going to go into the details and I think you know by the end


of this video you should know everything you need to know about GPL and you will


certainly be able to apply it in practice. So we've introduced you know


where we might want to use it I just want to now have a look at sort of like an overview


of GPL and where we are going to use it in this video. So GPL is you can use it


in two ways you can use it to fine-tune a pre-trained model so when I say


pre-trained I mean a transformer model that hasn't been fine-tuned specifically


for semantic search so BERT based case for example would be a pre-trained model


from Huggy Base Hub you take that and you can use GPL to fine-tune it for


semantic search you'll get okay results with that. The better way of using GPL is


for domain adaption so domain adaption is where you have you already have a


semantic search model so for example what we are going to look at we have a


model that has been trained on MS Marco from data from 2018. This model if


you give it some questions about COVID-19 because 2018 was pre-COVID it


it really kind of struggles with even simple questions you ask you a simple


question about COVID-19 and it will start returning you answers from about


like Ebola and flu rather than COVID-19 because it's confused it doesn't know


what you're it's never seen that before. So this is a typical issue with


sentient transformers they're very brittle and they don't adapt new domains


very well whatsoever. So GPL is best used in these sort of scenarios where we want


to adapt a model to a particular domain okay now as I said our example is going


to be COVID we were going to teach this model to understand COVID even though


it's it's never seen anything about COVID before. So let's have a look at how


GPL actually works. So at a very high level GPL consists of three data


preparation steps followed by one fine-tuning step so the data


preparation steps are the steps that produce the synthetic data set that we


then use in that fine-tuning step because that fine-tuning step is actually a


common supervised learning method. The unsupervised if you want to call it that


part of this method is that we are actually generating the the text or the


data that we'll be using automatically you know we don't need to label it


ourselves. So these three data preparation steps are the key to GPL


they are query generation which is where we create queries from passages so like


paragraphs of text. The second is negative mining so that is a information


retrieval step where we retrieve similar passages to our positive passage which


is a passage from before that do not actually answer our query or we assume


they do not answer our query and then the pseudo labeling step kind of cleans


up the data from those two steps from those two earlier parts and use the


cross-encoded model to identify the similarity scores to assign to our


passages and query pairs that we've generated already. So you can see that


process here we start with the P plus which means positive passage at the top


we pass that through a query generation model we use T5 for this and that


generates queries or questions. That's passed to a dense retrieval step the


negative mining step which will hopefully return things that either


partially answer or share many of the same words with our query or positive


passage and therefore similar but do not actually match to our query. And then we


pass all of those the query, the positive and negative through to our studio


labeling step which is the cross encoder step. Now the cross encoder step is going


to produce a margin score so that's almost like the difference in similarity


between our query positive pair and our query negative pair. We'll explain all


this in much more depth later on I assume right now it's probably quite


confusing so no worries we will go through all of this. So as you probably


notice each of these steps requires a user a pre-existing model that's been


fine-tuned for each one of these parts. Now we don't need to have fine-tuned this


they're general models they don't need to have been specially trained for that


particular task with this particular domain okay. These models are generally


quite good at adapting to new domains which is why we're using them. So let's


dive deeper into what I've just described and begin have a look at the


query generation step of GPL. So as I mentioned GPL is perfect for those


scenarios where we have no labeled data but we do need a lot of unstructured or


unlabeled data okay so we need a lot of text data that is not labeled that could


be text scraped from web pages from PDF documents from anywhere where you can


find text data. The only requirement inside this text data is actually


relevant to your particular use case e.g. is in domain. Now if we consider the


case of maybe we have a use case where we need to build a semantic search


retrieval model for German finance documents for the in domain of that we


might consider German finance news articles or German finance regulations


okay they would be in domain but other documents like Pytorch documentation or


English financial documents they are not in domain at all. It's almost like if you


imagine you're studying for an exam in one topic like biology and you start


studying some chemistry papers for that biology exam maybe some of that chemistry


might be relevant a crossover of your biology but not much you're never


actually going to do that if you want to pass your exam at least. Now in our


examples in this video we're going to be using the CORD 19 data set which is a


set of papers that I don't think all of them are specifically about COVID-19 but


I think most of them at least mention COVID-19 and there are a lot of files


here. So let's have a look at the code we use to actually download that data. Okay


so we need to find the CORD 19 data set from the Allen Institute of AI so we


this website here so it's not available at least when I checked it wasn't


available on Hugging Face data sets or anywhere like this so we need to pull it


manually. So I'm just going to download that we create this tar file and


essentially I'm not going to go explain all of this on to be quick through this


little bit but you will be able to find the link to this in the description so


you can just run this script but everything will be stored within this


document parses PDF JSON file okay which is just going to be a load of JSON


files there's a lot of them three three hundred just under three hundred and


forty thousand of them and they're gonna be named like this and they're gonna


look like this okay so they each have like these paragraphs in there which


we're gonna pull in and use in our examples. So once we've actually


downloaded all the test data we're gonna move on to the query generation code so


we have our CORD 19 data we're going to read them from here I use the generate


function here so we're going through getting the text and we're just yielding


the passage so this is one passage at a time so one paragraph takes that time


that I'm pulling in and I'm using a generate function here to do that now


just be aware that are do a lot of duplicates in this data set so what I've


done is create this passage dupes set to just check every time we have a new


passage we just check is it already we already pulled in if so we skip move on


to the next one otherwise it's a unique passage and we can we can pull it in and


add it to that duplication check and then yield it so using yield here


because this is a generator function that we loop through one at a time so


basically we're going to iterate through that function and it will return as one


passage at a time which is what we're doing here I return two passages here


you can see that pretty long okay so there is a lot of text data in there


we're probably going to cut a lot of it out when we're feeding into our models


that's fine we just want to see how this method works we want to make everything


perfect so what we're gonna do is use this model here so this is a a t5 model


trained on data from MS Marco from 2018 so it's pre-covid it doesn't know


anything about COVID we're using Hug & Face transformers here autotokenizer


and then automotive sequence to sequence language modeling and one thing just to


be very aware if you have a CUDA enabled GPU definitely try and use it so here


we're just moving that model to CUDA to our GPU it will make things much


quicker this set otherwise on CPU will take a very long time so it's best not


to if if you can it's best not to do that I think for me on the Tesla v100 I


think this took from maybe like one or two hours to do for 200,000 examples


that was just a fair few so here's just an example of what we're going to do here


so I'm taking one passage we are going to tokenize the passages to create our


inputs and now inputs will include the input IDs and attention masks we need


both of those we feed them into our model and this generates three queries


from that one other thing that you really should make note of here is I'm


again I'm moving those tensors to CUDA if I can okay if I move my model to CUDA


already these also need to be on CUDA and let's have a look at what we


produce from that so this is sometimes it's good sometimes it's not good it


really depends I think this is it's a kind of relevant example it's not


perfect because we're using the queries are about SARS here rather than COVID


even though we're talking about COVID obviously the model doesn't know that so


this generative step or query generation step is very good for out-of-domain


topics because for example if this said COVID-19 I think there's a good chance


that this would say where is COVID-19 infection found okay because it does a


lot of word replacement okay so it's seeing like your your answer basically


and it's just kind of reformatting that answer into a question about the answer


that's a lot of what this this t5 model is doing so it can be quite good and in


some cases it can be less good okay but that's fine because it's it's bad and


nothing we really do need some sort of label data here and in most cases it


works fairly well so here we get some we get some okay examples we have SARS or


COVID but otherwise it's fine so what I'm doing here is I'm creating a


new directory called data where I'm going to store all of the synthetic data we


create importing TQDM for the progress bar setting a target of 200,000 pairs I


want to want to create batch size increase that as much as you can that's


gonna make it quicker but obviously it's restricted depending on your hardware


and just specify number of queries and we're going to do this all in batches to


make things a little bit faster and then we reinitialize that that generator


that's going to return as our passages because we've already run through like


three of them already so I just want to restart it essentially so we're gonna go


through I'm using this with TQDM as progress here because this is a


generator we don't know how long it is so it's not gonna give us a good


progress bar if we just run TQDM on passages so I'm specifying the total as


a target so total number of steps is the target that I have up here the 200,000 so


I'm gonna increase the count by one with every new passage that we generate


queries for in fact I'm going to increase it by three I think yeah


because we create three queries per passage so we do do that and once our


count is greater than or equal to the target 200,000 we stop the whole process


here for the passage batch I'm appending a passage so the passages that we have


from here I'm just replacing any tab and newline characters to keep it clean for


when we're writing the data later on more than anything because we're going


to be using tab separate files we're going to encode everything in batches so


once we once the length of the batch is equal to the batch size that we specify


up here so 256 we're going to begin encoding everything so we tokenize


everything in the batch and then we generate all of our queries again


generating three for each passage that we have and then another thing is that


we need to decode the queries that we've generated to human readable text because


the next model we use is going to be a different model so it cannot read the


same tokens that this model outputs so we need to decode that back to human


text and then we're going to loop through all of the decoded outputs which


is one query per line and we have to consider this there's actually three


queries per passage so we have maybe five passages here and that means we have


15 queries on the other side so we need to consider that when we're pairing those


back together so we use this passage IDX which is the integer of I divided by a


number of queries so imagine you have so we have for passage 0 we are going to


have queries 0 1 & 2 okay 0 1 & 2 you divide those by 3 okay and all of them


are going to be less than 1 okay so so 2 is the highest one 2 divided by 1 0.66


if you take the integer value of that it's going to become 0 which maps to the


the passage on this side right okay so that maps to passage number 0 so that


means 0 1 2 all mapped to passage 0 then we do the next one so 3 4 5 and that's


going to map to 1 okay so that's how we're mapping our generated queries back


to our passages and then for each one of those we just append it to line here


we're using a tab to separate them because we're just going to write them to


file and we increase the count, refresh the batch every time we've


been through a batch and update the progress bar okay so updating it


actually updating it by 3 here no sorry updating it by the size of the decoded


output so if we had five passages that would be 15 because we have three


queries per passage okay now we want to write those query passage pairs to file


which we do like that okay so that's the query generation set and it's a lot in


there but from this we have we now have our query passage pairs now it is worth


noting that like we saw before query generation is not perfect it can


generate noisy imperfect like super nonsensical queries and this is where


the GPL improved upon GenQ if you know GenQ we covered it in the last chapter


and GenQ just relies on this this query generation step GPL later on we have


this pseudo labeling step which kind of cleans up any noisy data that we might


have or at least cleans up to an extent which is really useful so that's great


we've finished with our current generation step and now we're ready to


really move on to the the next step of negative mining which I think is


probably one of the most interesting steps so now if you think about the data


we have we have our queries and the assumed positive passages or the you


know the positively paired query passages that we have in a moment now


suppose we fine-tune our sentence transformer or bind coder with just


those our sentence transformer is just going to learn how to put things


together so you can learn how to place these queries and passages in the same


vector space and yes okay that will work to an extent but it's not the


performance is not great and what we need is a way to actually find negative


passages so there was a paper on rocket QA and they found that when or they had


this really cool chart that I liked where you have your performance or model


performance when they've trained it with hard negatives so negative samples are


quite hard for the model to to figure out that it's negative because it's very


similar to your positive and we'll explain more about hard negatives later


on and where they train the model without any hard negative so just the


positives and it's very clear that models perform better when there are hard


negatives it's almost like you can think of it as making your exams harder okay


the people that pass the exams will now will have studied even harder than they


would have in the past because you've made the exam harder they need to study


harder and as a result of that they will be better because of that their


knowledge will be better because the exam was harder okay they had to try


harder to pass it it's very similar in why we include negatives it makes the


training process harder for our model but at the same time the model that we


output from that is much better so to actually get these natives we perform


what's called a negative mining step now this negative mining step is used to


find highly similar passages to our positive passages that are obviously


very similar but are not the same okay so we are assuming that these very


similar passages are maybe they talk about the same topic but they don't


really answer our query okay we're assuming something like this we don't


actually know we're assuming this so we're performing a semantic search


information retrieval step to actually return these possible negatives and then


we can use them as negative training examples so now what we will have is our


query positive passage p-plus and assumed negative passage p-minus and


the result of this is that our model is going to have to learn very nuanced