Using Semantic Search to Find GIFs

You see here we have the image tag just putting that in and then we have the


Using Semantic Search to Find GIFs

description. Description pretty accurately describes what is happening


Using Semantic Search to Find GIFs

in the GIF. Okay now we have those duplicates or duplicate descriptions.


Using Semantic Search to Find GIFs

Let's have a quick look at those as well. So these are a few of those duplicates


Using Semantic Search to Find GIFs

and we can see that they are all the same GIF as expected and they just have


Using Semantic Search to Find GIFs

different descriptions. But the descriptions are all pretty accurate.


Using Semantic Search to Find GIFs

They're not, you know, they don't not describe the GIF. So in this case keeping


Using Semantic Search to Find GIFs

these duplicates makes sense because we just simply have multiple descriptions


Using Semantic Search to Find GIFs

that are going to point to the same GIF. All of those descriptions are accurate


Using Semantic Search to Find GIFs

so it's not really an issue. So we have our data set. Now let's just have a quick


Using Semantic Search to Find GIFs

look at our graph here. So the first thing we need to do here is initialize


Using Semantic Search to Find GIFs

our retrieval model and our vector database. And as you can see on the left


Using Semantic Search to Find GIFs

here we are going to be indexing our data using our retrieval and vector


Using Semantic Search to Find GIFs

database before we begin querying anything. Basically we're just going to


Using Semantic Search to Find GIFs

take all that data putting it into our vector database. So the first thing I'm


Using Semantic Search to Find GIFs

going to do here is initialize the retrieval model which is here and I'm


Using Semantic Search to Find GIFs

going to initialize this sentence transform model. If you don't know


Using Semantic Search to Find GIFs

anything about sentence transformers there's a lot of videos on my YouTube


Using Semantic Search to Find GIFs

channel and articles on Pinecone that cover these in a lot of detail. So I'd


Using Semantic Search to Find GIFs

recommend having a look at those because they are really interesting. Now one


Using Semantic Search to Find GIFs

important thing here to note is that our model is going to be outputting vectors


Using Semantic Search to Find GIFs

with dimensionality 384. Okay I think earlier I said 374, it's 384. That's


Using Semantic Search to Find GIFs

important we need to know that when we're initializing our vector database


Using Semantic Search to Find GIFs

which we can do like this. So we import Pinecone. For the API key here you do


Using Semantic Search to Find GIFs

need to go to app.pinecone.io to get a free API key and you just put it in here.


Using Semantic Search to Find GIFs

And then with that we can initialize our connections Pinecone. Then we all


Using Semantic Search to Find GIFs

we need to do is pass an index name. So index name can be anything you want. Just


Using Semantic Search to Find GIFs

make sure it makes sense. So for me this is a GIF search index so I am going to


Using Semantic Search to Find GIFs

create that index with the dimensionality 384. Okay so that's what we got from the


Using Semantic Search to Find GIFs

retrieval model earlier. The metric is also important. This should align to what


Using Semantic Search to Find GIFs

the sentence transformer retrieval model has been fine-tuned to work with. And


Using Semantic Search to Find GIFs

then we connect to the index we've just created. Okay so once we've done that we


Using Semantic Search to Find GIFs

move on to actually indexing. Okay so initialize the vector database, initialize


Using Semantic Search to Find GIFs

the retriever. Now use the retrieval model to create our embeddings or the


Using Semantic Search to Find GIFs

vectors that represent all of our GIF descriptions. And then we insert all


Using Semantic Search to Find GIFs

those into Pinecone. So we do that in this loop. It's pretty simple. We do it in


Using Semantic Search to Find GIFs

batches of 64. Here I'm extracting a batch from the data frame that we have. I


Using Semantic Search to Find GIFs

generate embeddings for that batch. We retrieve the metadata for that batch. So


Using Semantic Search to Find GIFs

the metadata is going to be in a format. Do I have something here? Maybe I don't.


Using Semantic Search to Find GIFs

So the metadata will be in the format. Select description, description, and then


Using Semantic Search to Find GIFs

we have a description of what is happening in there. And then also the


Using Semantic Search to Find GIFs

URL. And that will go to the URL. Right, pretty simple. That's our metadata. And we


Using Semantic Search to Find GIFs

have one of those for each record or item. And then we create our IDs. So the


Using Semantic Search to Find GIFs

IDs need to be unique. There needs to be strings. And that's what we're doing here.


Using Semantic Search to Find GIFs

Okay so we're just going through the IDs. It's simple. 0, 1, 2, 3. But obviously


Using Semantic Search to Find GIFs

there's a string. And then we put all that together. Okay and we insert these to


Using Semantic Search to Find GIFs

Pinecone. So we do that in batches of 64. And then at the end here I'm just checking


Using Semantic Search to Find GIFs

that we have all the vectors in the index. Which we do have a hundred and


Using Semantic Search to Find GIFs

twenty five thousand or just over. Okay cool. And we can also see the index


Using Semantic Search to Find GIFs

fullness. So we can go up to about a million vectors on the free tier of


Using Semantic Search to Find GIFs

Pinecone. So here you can see we have plenty of space left which is obviously


Using Semantic Search to Find GIFs

pretty useful. So with that we can move on to the right of our chart here which


Using Semantic Search to Find GIFs

is the querying step. So querying is what we're going to do all the time. So the


Using Semantic Search to Find GIFs

indexing we do once. Unless we're sort of adding more data and then we might do it


Using Semantic Search to Find GIFs

again. Otherwise querying is the main task of our pipeline. Every time a user


Using Semantic Search to Find GIFs

makes a query, searches for something, we're going to be going through this pipeline.


Using Semantic Search to Find GIFs

So to do that again it's pretty much same as before. Query goes through to our


Using Semantic Search to Find GIFs

retriever model. That creates what we call a query vector. We pass that to Pinecone


Using Semantic Search to Find GIFs

and then we search for the most similar already indexed GIF description vectors


Using Semantic Search to Find GIFs

or context vectors. And then from there we return the most similar ones. Okay and


Using Semantic Search to Find GIFs

we within those records from Pinecone we also include the metadata and that


Using Semantic Search to Find GIFs

includes the URL to the original GIF. So then we can just use some simple HTML to


Using Semantic Search to Find GIFs

display those GIFs to the user. And let's have a look at how we actually do that


Using Semantic Search to Find GIFs

in code. So I split into two steps. There's a search section or part which


Using Semantic Search to Find GIFs

is searching, encoding our query and also searching for the most relevant context


Using Semantic Search to Find GIFs

vectors. And that's what we're doing here. So we encode with our retriever, we query


Using Semantic Search to Find GIFs

and we are going to return the top ten most similar contexts there. And yeah we


Using Semantic Search to Find GIFs

just append that to this list here. Okay you can see here we're getting the


Using Semantic Search to Find GIFs

metadata out as well. It's also important. So here include metadata needs to be


Using Semantic Search to Find GIFs

true. Otherwise we're not going to be turning any metadata and we can't extract


Using Semantic Search to Find GIFs

those URLs. Okay and I'm just putting that in a search GIF function there. And


Using Semantic Search to Find GIFs

then we have this display GIFs. So all this is is the really simple HTML that


Using Semantic Search to Find GIFs

we are displaying. So we just have these div elements inside here we have a


Using Semantic Search to Find GIFs

figure and our image with the URL source that we have. Okay and then to do that


Using Semantic Search to Find GIFs

or to perform our search we just do search GIF, dog being confused in this


Using Semantic Search to Find GIFs

case, and we display them. Okay and we can see that we're getting all of these


Using Semantic Search to Find GIFs

GIFs where a dog is confused which is I think pretty cool. And we have loads of


Using Semantic Search to Find GIFs

examples here. I just went through a load them to see what we get. And yeah.


Using Semantic Search to Find GIFs

Oh this one quite specific. You can get really specific with this as well. So


Using Semantic Search to Find GIFs

fluffy dog being cute and dancing like a person. And then we get this one up at


Using Semantic Search to Find GIFs

the top here. So yeah I think that is pretty cool. Now if we want to replicate


Using Semantic Search to Find GIFs

the sort of app, the Streamit app that you saw before, you can do one if you


Using Semantic Search to Find GIFs

just want to test it you can. But to create this app it's super simple.


Using Semantic Search to Find GIFs

Obviously we're using Streamlit and this is all we do. I'm gonna zoom out a bit


Using Semantic Search to Find GIFs

so it might be kind of hard to read on your screen but it's on to kind of show


Using Semantic Search to Find GIFs

you the whole code. And again you can just download this anyway so it shouldn't


Using Semantic Search to Find GIFs

really be an issue. This code is slightly outdated actually. So this model here is


Using Semantic Search to Find GIFs

not using the MPNet model anymore. It's actually using the MiniLM model. It's


Using Semantic Search to Find GIFs

just a smaller model. And yeah all we do we initialize PineCone, initialize our


Using Semantic Search to Find GIFs

retriever like we did in those notebooks. We have our HTML here. It's actually the


Using Semantic Search to Find GIFs

same as in the notebooks again. And then we just write our app. So we have AI


Using Semantic Search to Find GIFs

powered GIF search. What are you looking for? This is a text input that passes


Using Semantic Search to Find GIFs

into query. Whenever query is not empty we will begin to search. Okay so we


Using Semantic Search to Find GIFs

encode as we did before the query to create our query vector. We retrieve the


Using Semantic Search to Find GIFs

most similar context and then we extract the URL metadata from those. And then we


Using Semantic Search to Find GIFs

display them. And that's literally it. It's super simple and we get this really


Using Semantic Search to Find GIFs

cool GIF search app from that. So that's it for this video. I hope this has been


Using Semantic Search to Find GIFs

interesting. So thank you very much for watching and I will see you again in the


Using Semantic Search to Find GIFs


How-to Use HuggingFace's Datasets - Transformers From Scratch #1

Hi, welcome to this video. This is the first video in what will be kind of like a mini-series on


how we can train a transformer from scratch. So a lot of you have been asking for this in one way


or another. So we're just going to run through kind of everything I think you need to know


to actually just build your own transformer. So there are a few different parts to this,


and that's why I'm doing it in a series because otherwise it would be a super long video.


So we're just going to break it down into a few different parts. So the first video is going to be


on getting our data, and that's what you're watching now. So we're going to learn how to use


the Hugging Face Datasets Library, which I think is very good actually. So we'll take a look at that.


In the next video, we'll have a look at actually training the tokenizer with that data. And then


in these bits here, so these three parts, they might just be one video. And I'm not sure I'm


going to see how it flows. So there will probably be maybe one or two videos. Let's see.


So we'll move on to getting our data over in Python. Now, when it comes to getting data to


train a transformer model, we are pretty much spoiled for choice. All we really need is


unstructured text data, which of course, there's a lot of that on the internet.


Now there's one dataset in particular that I've noticed that is very good called OSCAR.


And that is a multilingual dataset. It has like hundreds of languages, I think,


like really loads. So we're going to go ahead and use that. Now to download the dataset or the OSCAR


dataset, we will be using a Transformers library called Datasets. So if you do need to, you can


pip install that. So you just pip install Datasets. And once we have that, we just want to import


datasets. Now, what we can do if we want to just view all of the datasets that are available to us


within our notebook here is we can write datasets.listDatasets.


And let's just print out how many we have because there's quite a few. So at the moment, there's


965 of those. So I'm not going to print all of them out, but let's just have a look at


a few of those so we can see what we actually have here. So here's just five. So it's


alphabetically sorted. So we have five datasets all beginning with A. Now, one of these is called


OSCAR. So we can write OSCAR in or DS. We see true, but we can't really get that much information


from within Python. So what we can do is head on over to this website here, which is the Hugging


Face datasets viewer, which is really cool. So what we do is we go over here on the left and


we have we can search for a dataset based on tags. Or what we'll be using is the actual dataset name.


So we can just go through and see all these datasets. There's loads of them. Now, if I scroll


down quite a lot, you can also type in at the top, you will find or I should find OSCAR here.


Now, we search for OSCAR and then we also get this subset here. This is another important thing. So


OSCAR has all these different subset of languages within its dataset. And these are all of those.


Now, if you want to know what each of those are, because we just get a letter here, which is not


really that useful. We can go over here. So this is oscarcorpus.com. And I believe we click here.


Okay, so we scroll down to here and we have a big list of all of the languages here. So we have


the language and then we have the AF here, which is the first one. So the first one we know is


African. Now, if we scroll down, I'm going to go, I don't, I only know English, but my girlfriend


is Italian and she's going to come along and hopefully tell us that the model works at some


point. So we're going to be using Italian because that's literally the only choice I have other than


English. And that's kind of boring. So we're going to go with this one. So,


we need to have, we need to search for the one with IT at the end. We come here and we can just


type in, I think, or maybe we can't. There we go. So click that and we are not actually allowed to


view it because it's too big. I didn't realize that. Let's go with, I think Latin you can view.


Yeah. So obviously this is in Italian, this is Latin, but you can see here we have the ID and we


have text and this is the data set that we're going to be using. So let's go back over to,


let's copy this and we'll go back over to our code.


Now, what we do is we're going to be loading that data set. So we're going to go to the


now what we do is we're going to be loading that data set or downloading the data set into this


data set variable. We want to write data sets, load data sets. And here we want to write Oscar,


which is the data set name. And then we also want to include the subset. So our subset is not pasting.


That's fine. It's unshuffled, deduplicated, IT.


So, here we go. So I already have it downloaded, I think. So that's not going to download it again,


but you should get a loading bar. This is your first time downloading this data set.


And that might take a little bit of time. Okay. So that has loaded for me now. So I can


do this and we can see that we have this data set dictionary object. And inside there, we have this


one item. So sometimes you have like training data and testing data. We just have training data here.


So we have train and then inside that we have our data. So we have this many samples, which is


28.5 million samples. I'm not going to use all of them because it will just take a very long time.


And we have the two features that we saw before. So we have the ID and then we also have the text,


which is what we care about. Now, if we wanted to just have a look at one of those,


we could write the train zero like this.


And we see our data. Okay. Now, that's good. But when we're training our tokenizer, we're going to


want to read these in from file rather than keeping them in memory. So we can see that


we want to read these in from file rather than keeping them in memory. So what we're going to do


is first I'm going to import TQDM because this can take a little bit of time. So I want to have


a loading bar so that we can actually see what is happening. And from there,


we want to initialize a list, which is going to contain our text data. And what I'm going to do


is I'm going to loop through all of this data, format it in a way that we can then save it to


file that we would expect for the tokenizer. So it essentially needs every sample to be on a new line.


And I'm just going to take, I think, 10,000 of those samples, put them into a file, and then


move on to save it and move on to the next file. So this is what the file count is for. I'm just


going to write something like Italian data zero, Italian data one, Italian data two, and so on.


So we're going to loop through all of our samples. So for sample in, and here I'm going to wrap it


in TQDM so that we can see the loading bar or the progress bar. And here we're just going to go


data set, train, so that will go through all of our samples. Now, we're going to be splitting


each sample with a new line character, which also means we need to remove any other new line


characters from our data. Otherwise, we're going to be splitting each sample into multiple samples,


which we don't really want. So we write sample equals sample. And in here, remember, we have like


ID. We have ID and text here. So we want to access the text specifically. And we're going to replace


the new line characters in there with just spaces, I think. Yeah. Then what we're going to do is


text data, append, sample. So that is going to add one sample to our text data list up here. Now,


what we want to do is say, if the length of that text data list


hits 10K, at that point, we want to save it to file.


So I'm going to write with open. I'm just going to call it Italian text, or just IT.


And I'll put in the file count. So file count.txt. We're going to be writing that.


And then we just write fp.write. We're using new line characters here. So we're just going to join


everything within our text data list like that. And we also want to just here include the encoding.


So utf-8. Now, once we've written that data, we don't want to keep all the current data.


Within the text data variable or text data list, because we have 10K, we want to sort of


reinitialize that list so that we start again. And then we print the next, or we save the next 10K


after that. So we want to write text data equals, it's going to equal an empty list.


And then obviously we just keep saving it with the current file count. We're just going to keep


overwriting ourselves. So we need to add one to the file count. Now that will save most of our data.


But on the final batch, so we have, what do we have?


So we have this many samples.


And if we take that, we are left with 2082 samples at the end there. So that means they will not save


because it will not reach the 10K on that final loop. So at the very end here, all we're going to


do is, I think we just copy this. And then we'll just save it. And then we'll just save it.


And then we'll just copy this. So we will copy that and yeah, that should be fine.


So that will save that final 2K that we have there. So I'm going to run that. I think it can take a while.


Obviously, I think it does take a while. So that's 20. That's kind of weird.


Yeah, it's going to take maybe 30 minutes.


So that's fine. So after we have done that, we move on to


what I'm going to cover in the next video, which is actually training our tokenizers.


Why are there so many Tokenization methods in HF Transformers?

Hi and welcome to the video. Today we're going to have a look at all of the different tokenization


methods or a few of them at least in HuginFace Transformers. Now I'm sure a few of you are asking


tokenization is pretty straightforward and I believe this as well. So why are there so many


tokenization methods? So on the screen right now you can see we have these five different methods.


Now in reality each of these actually does do something different but all of them are simply to


produce token IDs. Now for those of you that are new to tokenization and maybe Transformers we'll


just quickly have a look at the very basics of a tokenizer or the very basics for understanding


what each of these methods actually does. So tokenization in short is this. The process of


going from what we have up here which is our original human readable text so hello world we


also have this exclamation mark at the end there. Converting that original text into what we call


tokens. Now tokens can be well they can be a few different things. In this case what we see is


tokens built from words so each token represents a word or a part of the syntax so the exclamation


mark at the end. Now depending on what sort of tokenizer you're using you can build tokens from


completely different things. So you can build tokenizers from the bytes within the text. You can


do word piece encoding so in this case there's no good examples but say maybe we had the word


maybe we had the word something. Okay we can easily split this into I think probably three


different word pieces so we have some and then we have ing at the end that's a common part of a word


so that would be a word piece in itself and then we also have thing in the middle there.


So we can tokenize it doesn't have to be a single word for each token it can be a whole host of


different things and then we go from those tokens to the token ids which we see the bottom.


So in this case hello is being translated to 7592 the integer and then we have word and also the


exclamation mark as well. So that's the process that's what we're doing but how do we do that with


a Hugging Face Transformer. So we have these two files that our tokenizer is built from.


So these two here that's our tokenizer and when we build a tokenizer you if you've followed some of


my previous videos on building a tokenizer you will recognize both of these files and these are


the two steps. So the first the mergers.txt takes us from that original text here to our tokens down


here. So that's step one and then step two is where we go from those tokens that we previously


built in step one we process them through vocab.json and that produces our transformer


readable token ids which we see the bottom there. Now there are a few different tensors that we need


for feeding into our model. So with transformers so we've just seen how we build the input ids


or token ids. So that's the essential we need that for every transformer model. So token ids we also


have the attention mask. I'll just write mask for now. These are the typical ones that we


we would see. So the attention mask is typically a tensor containing ones and zeros. The ones


will correlate to the real tokens within our token ids tensor and the zeros correlate to


padding tokens in the token ids tensor. So we have the attention mask and then we also have the


token type ids or these you can call them segment ids as well. And segment ids are used


when we have multiple segments to our inputs. So we might have token ids and maybe we are doing


question answering. So question answering we would have if we're feeding it into BERT we would have


our question. I believe it's in this order so we would have the question and then in the middle


we'd have this separated token. I'll just write sep and then we would have the context that we're


getting the answer to our question from. Now in segment ids anything that belongs to our question


would be represented by a zero. Anything that belongs to our context would be represented by a


one. So they're the three key tensors that we would be using. And here's just a visualization of the


attention mask. So we have the real tokens represented by ones in the attention mask tensor


and then the padding tokens represented by the zero in the attention mask tensor. Now I think


I mean that's all we really need to do. A quick summary of token izers in transformers. Now


how does that correlate to what we're doing here? Well let's create a new cell


and let's take this as our first example.


So we have our text hello world and let's have a look at what happens when we use tokenizer,


tokenizer alone. Okay if we do this we see that we create our tokens. So


straight away we know that this method here our first method it does our tokenization


in the steps that we outlined before. So it doesn't do everything all at once it does them


step by step and you can probably guess that the next step from creating those tokens


is to convert them into token ids which we do there. Now it's a completely valid method.


This it works it's good but you'll see that it's pretty simple. Like what we have here we can't


create we can't create pytorch tensors or tensorflow tensors. There's no arguments for adding


padding or truncation which we almost always need and we also can't add special tokens.


So I mean it works it's fine but it's very simple. So maybe it's not the best thing if you


want to do all of that stuff automatically. Maybe you want to do it manually in that case you can


go ahead you can add your special token your padding your truncation manually without a problem


and then also convert those into pytorch tensors. So we can do that that's fine.


The maybe easier method is if we go ahead and use encode. So if we look at encode here


you can see that we have these two tokens. So we have the same as what we've got up here the 7592


to 999 so that is our text tokenized or converted into token ids.


Then we also have this 101 102. Now if you don't know what those are it's fine.


They are basically special tokens that BERT uses to indicate the start sequence for the 101


or the end of a sequence for 102. And there's also another special token that we'll see in a minute


which is zero which is the padding token that BERT uses. Now if we were to use this method for


actually building a tensor for pytorch for BERT we would probably write something like this.


So we'd set the max length equal to 512. We'd set padding equal to the max length.


And then we would make sure that we return


pytorch tensors like so. Right and then we see we get this big pytorch tensor with all these zeros


and they are padding tokens right. And that goes up to a length of 512 which is the correct size for


BERT base. So that makes sense I think. Let me just reset this.


So that makes sense I think. Let me just restrict how much of that we're seeing.


Maybe a little bit less. Let's go to 10. Okay.


So that's ENCODE. Now up here we also have ENCODE plus.


So let's try and see what this is. So we'll see up here we just got our input IDs. So we refer


back to here. We have our token IDs or input IDs but we don't have the mask or the segment


IDs which we also need. And that's a limitation of the ENCODE method which gets fixed using the


ENCODE plus method. So if we run that we see that instead of getting a single list we return


dictionary that contains the input IDs or token IDs, the token type IDs or segment IDs and the


attention mask. So that's straight away it looks a lot better. Now we can also use all the same


arguments that we use in ENCODE. So let's change to ENCODE plus.


And we'll remove that for now. I'll add it back in a minute. You can see it now. Okay we have input


IDs and we'll go down. We might have to go down a little bit. And then we have token type IDs.


We have our token types here. Now this is just zeros because you don't have two sequences in


there. But if we were to pass two sequences we would get the zeros and ones. And then we also


see we have the attention mask. So that's three methods. We have the convert tokens and the tokenize


or tokenize and convert tokens to IDs. Then we have ENCODE and ENCODE plus.


Now you may have guessed already from the name but we also have this batch ENCODE plus. Now batch


ENCODE plus allows us to do the same as what we do with ENCODE plus but for batches of sentences.


So if we had... Let's go down here. Let me just remove that.


All right so let's take... Let me take this for now.


Right so let's create a text list.


And in here we're going to have text and I'm going to add another item as well. So


let's create a text list. And in here we're going to have text and I'm going to add another


item as well. So hello world again. Okay now in here if I were to write ENCODE plus and text list


we see that we get this pretty weird output that doesn't... I mean just doesn't look right. That's


because it isn't right. We can't pass a list to a list that's not in the list. So we're going to


pass a list to the ENCODE plus. It won't work. We have to pass each string one at a time. And


actually what we can see here is we have these 100 tokens. 100 is the unknown token and that's


because we're passing two objects in a list. String objects in list and the tokenizer is reading


the objects as a whole and saying I have no idea what this is. So it's just giving it a unknown


token. So we can't use ENCODE plus. Instead we use batch ENCODE plus like that. And now we see that


we get... We don't only get one of our tensors but we also we get two of our tensors. So we get


an array for each one of these. So if we... Let me just write this.


Okay. If we go token ids we can access each one of those tensors because it's in a dictionary. So


you write input ids and let's have a look at the shape. Sorry. So I need to... Let me just return


tensors. So we can only use the shape method when we have tensor. Also we need to do that. So let


me take that. So the reason we were getting that error is because our two arrays are of different


lengths and we can't create a tensor where we have differently sized rows. If that makes sense. So


you'll see in a moment if I change the max length to 10. So it's not huge. So we can see here that


we've added in our first row we've added five padding tokens and in the second we've added four


to make them both the equal size. So we can actually create a tensor from them. So let's now


go for shape and we see that now we have our two strings that have both been tokenized and


converted into a list or a tensor row of 10 values. So that's the batch in code plus. We can add a


huge number of strings into that. We just need to add them as a list. And that leads us on to our


final method which is this tokenizer. So you see here we've been using tokenizer and then followed


by a method within that class or that object. This time we're just using the class or calling


that class directly. So tokenizer and let's put text. OK. And I mean if we look at this it looks


like it's doing the same as encode plus. So if we compare it to encode plus exactly the same output.


Now what if we convert this into text a list. OK. So now we're getting the same output as we got


from batch encode plus. So what tokenizer doing or calling tokenizer directly is doing it's looking


at the data type of the input that we're feeding into it. Is the data type a string as is the case


with text or is the data type a list as is the case with text list. It looks at either of those


and it says OK if it's a string I'm going to call the method encode plus. If it's a list I'm going


to call I'm going to call the method batch encode plus. So that's all that tokenizer is doing. So


generally tokenizer is usually the way to go if you're not sure whether you've got batches or


strings coming through it can be very useful to use tokenizer. But that's that's pretty much all


I wanted to to cover for this video with with tokenizer as well. It's worth noting we can use


the same parameters. So if we take these we can we can use all of these as we did with encode


and encode plus. That's it. So there are all the tokenization methods or the the main ones


in transformers. Now I'd imagine there's probably more that I'm not aware of. And if you're aware


of those let me know in the comments below that would be pretty interesting to see more of those.


But they're probably the main the five main ones and I've seen a few questions on those before so


I thought it'd be worth covering those. And I was also curious myself as to what the actual or the


specific differences are between each one of those. So that's it for this video. Thank you very much


Every New Feature in Python 3.10.0a2

Hi, we're going to go through the second alpha version of Python 3.10 today.


Every New Feature in Python 3.10.0a2

So it was released at the beginning of November


Every New Feature in Python 3.10.0a2

and with it we can already see quite a few new features.


Every New Feature in Python 3.10.0a2

So we're just going to go through those and it seems honestly pretty interesting.


Every New Feature in Python 3.10.0a2

So I think the first thing we're going to go through is the further extensions to the type annotations.


Every New Feature in Python 3.10.0a2

So on that front there are three main new features.


Every New Feature in Python 3.10.0a2

The postponed evaluation of annotations, the type union operator and the type alias annotation.


Every New Feature in Python 3.10.0a2

So for those of you that are not aware, we have the timeline of the type annotations with each new Python release here.


Every New Feature in Python 3.10.0a2

So in version 3 we had the initial implementation of function annotations


Every New Feature in Python 3.10.0a2

and then on and on since then there's more has been added.


Every New Feature in Python 3.10.0a2

There are a few different ways to do different things up until version 3.9 where everything was aligned


Every New Feature in Python 3.10.0a2

and they added the typing module syntax to the standard collections.


Every New Feature in Python 3.10.0a2

So this extension of the typing annotations has been continued with version 3.10.


Every New Feature in Python 3.10.0a2

The first one is the postpone evaluation of annotations and this is the point


Every New Feature in Python 3.10.0a2

that Python will actually evaluate the type annotation in our code.


Every New Feature in Python 3.10.0a2

So typically this would be done at function definition time meaning that


Every New Feature in Python 3.10.0a2

they're evaluated line by line in a top-down fashion and obviously that kind of makes sense.


Every New Feature in Python 3.10.0a2

But at the same time this can cause issues because if we for example define a type at the top


Every New Feature in Python 3.10.0a2

or refer to a type near the top of our code that actually doesn't get defined until later on.


Every New Feature in Python 3.10.0a2

I mean even though it still makes sense it's just a forward reference that will cause problems.


Every New Feature in Python 3.10.0a2

So what they've done is postponed evaluation until later on.


Every New Feature in Python 3.10.0a2

So this also speeds up the module imports very slightly and also prevents this issue of forward referencing not working.


Every New Feature in Python 3.10.0a2

So that's pretty much it for the postpone evaluation of annotations. It's pretty straightforward.


Every New Feature in Python 3.10.0a2

The next type union operator. So let me show you what we had to do before.


Every New Feature in Python 3.10.0a2

So let's go back to Python 3.9 here. So say for example we have a function


Every New Feature in Python 3.10.0a2

and we can take either an integer or a float into that function.


Every New Feature in Python 3.10.0a2

Now before what we would have to do or at least in 3.9 we would have to use the union like this.


Every New Feature in Python 3.10.0a2

And then we would pass a list of our two types. So int float like that.


Every New Feature in Python 3.10.0a2

I'm just going to output a float. You'll see why in a minute.


Every New Feature in Python 3.10.0a2

So here we're just going to return x times by 3.142.


Every New Feature in Python 3.10.0a2

OK so obviously with this it would make sense if x is either an integer or a float but nothing else.


Every New Feature in Python 3.10.0a2

So the way that we would do that before is using this union operator.


Every New Feature in Python 3.10.0a2

And with that union operator we'd have to import that from typing.


Every New Feature in Python 3.10.0a2

So like this. OK let me just save this. OK.


Every New Feature in Python 3.10.0a2

So now let's run it and see what happens. OK.


Every New Feature in Python 3.10.0a2

Sorry I need to do something here.


Every New Feature in Python 3.10.0a2

So we pass a for which is an integer. And then we'll just pass this as well.


Every New Feature in Python 3.10.0a2

So this makes sense. OK that's fine. It works as we would expect.


Every New Feature in Python 3.10.0a2

So what Python 3.10 has added here. Get back to this.


Every New Feature in Python 3.10.0a2

So we don't need to import this anymore.


Every New Feature in Python 3.10.0a2

And instead of using this union operator.


Every New Feature in Python 3.10.0a2

It was a really nice new syntax slightly bitwise or operator.


Every New Feature in Python 3.10.0a2

And we just use that. So int or float. And this will work in just the same way.


Every New Feature in Python 3.10.0a2

And that didn't work before before we would get a syntax error.


Every New Feature in Python 3.10.0a2

So we run a syntax 3.9. OK unsupported operator. Sorry type error.


Every New Feature in Python 3.10.0a2

So that's the other new addition to typings.


Every New Feature in Python 3.10.0a2

And then the final one is the type alias annotation.


Every New Feature in Python 3.10.0a2

OK so the current issue with the type aliases is that we if we are forward referencing.


Every New Feature in Python 3.10.0a2

Another type that hasn't been defined yet we would use a string.


Every New Feature in Python 3.10.0a2

So for example if we are importing this type from another module.


Every New Feature in Python 3.10.0a2

But the module hasn't been imported yet. We might want to reference that type.


Every New Feature in Python 3.10.0a2

And we would do that just using a string like this.


Every New Feature in Python 3.10.0a2

And this is fine but obviously it's just not very explicit as to what we're doing here.


Every New Feature in Python 3.10.0a2

So could write something like this.


Every New Feature in Python 3.10.0a2

With type alias as a type.


Every New Feature in Python 3.10.0a2

We also want to return type alias. OK return name there.


Every New Feature in Python 3.10.0a2

So if we run that it's fine.


Every New Feature in Python 3.10.0a2

OK and we can call it and we say.


Every New Feature in Python 3.10.0a2

Pass the name hello. Works fine.


Every New Feature in Python 3.10.0a2

But the issue is here that it's not clear that this is obviously we've written type alias here.


Every New Feature in Python 3.10.0a2

But if you get another name it's not very clear that this is actually a type alias.


Every New Feature in Python 3.10.0a2

So what has been included now is switched over to Python 3.10.


Every New Feature in Python 3.10.0a2

OK just rerun this.


Every New Feature in Python 3.10.0a2

OK so works.


Every New Feature in Python 3.10.0a2

And so what we now have in Python 3.10 is a type alias type.


Every New Feature in Python 3.10.0a2

So if we do from.


Every New Feature in Python 3.10.0a2

From type extensions.


Every New Feature in Python 3.10.0a2

Import type alias.


Every New Feature in Python 3.10.0a2

OK.


Every New Feature in Python 3.10.0a2

And now we make it explicit that this.


Every New Feature in Python 3.10.0a2

Is in fact a type alias and not a string.


Every New Feature in Python 3.10.0a2

OK and then we can run this again.


Every New Feature in Python 3.10.0a2

OK that works but now it's just more explicit that yes.


Every New Feature in Python 3.10.0a2

This is not string. This is in fact a type alias.


Every New Feature in Python 3.10.0a2

So that's another addition and that's everything that has been added in terms of the.


Every New Feature in Python 3.10.0a2

Type annotations.


Every New Feature in Python 3.10.0a2

And obviously I mean this is not like a huge massive change but it's I think it's very cool regarding the past changes in type annotations.


Every New Feature in Python 3.10.0a2

To see that the developers are doubling down on enhancing.


Every New Feature in Python 3.10.0a2

These features.


Every New Feature in Python 3.10.0a2

Obviously a big strength of Python comes from its ease of use and the lack of a steep learning curve.


Every New Feature in Python 3.10.0a2

And one of the reasons for this is the lack of need to actually explicitly define types in our code.


Every New Feature in Python 3.10.0a2

Unlike for example C where you are explicitly defining what every variable is and the type that it is.


Every New Feature in Python 3.10.0a2

Python we don't need to do that and that makes it a lot easier to get started which is great and that's really good.


Every New Feature in Python 3.10.0a2

But it's also a little bit difficult when it comes to actually reading more complex code bases.


Every New Feature in Python 3.10.0a2

If you if you go into a popular open source library and read through the code figuring out what is going into a function and what's coming out of it is really difficult.


Every New Feature in Python 3.10.0a2

But when we start to add.


Every New Feature in Python 3.10.0a2

Things like this.


Every New Feature in Python 3.10.0a2

It makes it incredibly easy.


Every New Feature in Python 3.10.0a2

So for example instead of.


Every New Feature in Python 3.10.0a2

This or actually maybe this is a better example.


Every New Feature in Python 3.10.0a2

Like here we know that X has got to be either an integer or a float and we know that it's going to return a float.


Every New Feature in Python 3.10.0a2

If we got rid of these obviously with this function it's really obvious.


Every New Feature in Python 3.10.0a2

But when you have something like some big machine learning library like TensorFlow or PyTorch and reading through that.


Every New Feature in Python 3.10.0a2

It's pretty confusing.


Every New Feature in Python 3.10.0a2

And without these types it's I mean it can get very very difficult.


Every New Feature in Python 3.10.0a2

But now if you look at for example parts of this transformers library which is another machine learning library they've started adding it adding in types and it makes it so much easier to read.


Every New Feature in Python 3.10.0a2

So for that reason I think that this these little differences or these little additions are very cool and really really good to see.


Every New Feature in Python 3.10.0a2

So on the other features that have been added we have a few as a new equal length flag for the SIP function.


Every New Feature in Python 3.10.0a2

So if we have two lists here and then we have another one.


Every New Feature in Python 3.10.0a2

Normally if we create a zip generator with both of these we'll see that there are no issues here.


Every New Feature in Python 3.10.0a2

And I mean it's kind of weird because neither these are not the same length.


Every New Feature in Python 3.10.0a2

So this doesn't cause any issues which I mean maybe that's fine but to me at least that's pretty weird.


Every New Feature in Python 3.10.0a2

What it does is actually just truncates the longer string sorry the longer list.


Every New Feature in Python 3.10.0a2

And I imagine a lot of developers have had a headache trying to figure out what has gone wrong because of this kind of strange feature.


Every New Feature in Python 3.10.0a2

So what they have added in Python 3.10 is a strict flag.


Every New Feature in Python 3.10.0a2

So rather than just allowing this to happen we can run it with a strict flag set to true.


Every New Feature in Python 3.10.0a2

OK so now we actually get a value error saying the argument 2 is shorter than argument 1 which is a nice I mean it's nothing crazy but it's a nice addition.


Every New Feature in Python 3.10.0a2

So I mean rather than blindly truncating mismatched data we actually can check now with the built in argument which is pretty nice I think.


Every New Feature in Python 3.10.0a2

And I think will save people a lot of time trying to figure out what what has gone wrong with their code.


Every New Feature in Python 3.10.0a2

So it's pretty useful. The next one is the new bit count.


Every New Feature in Python 3.10.0a2

So here we're actually counting the number of active bits in that integers binary value or binary representation.


Every New Feature in Python 3.10.0a2

So one so zero obviously it's just going to be this.


Every New Feature in Python 3.10.0a2

One the binary representation is this. Two this so we can see right for the first one there's zero ones in there.


Every New Feature in Python 3.10.0a2

For number one there's one one. Number two there's two ones.


Every New Feature in Python 3.10.0a2

So number three we would expect that to be two ones.


Every New Feature in Python 3.10.0a2

And which there is. OK so the final of the other features are the mapping for dictionary views.


Every New Feature in Python 3.10.0a2

So in Python we have three different dictionary views that we can use.


Every New Feature in Python 3.10.0a2

And that is a view for keys values and the dictionary itself or the dictionary items.


Every New Feature in Python 3.10.0a2

So if we create a dictionary here. Very simple one.


Every New Feature in Python 3.10.0a2

Yeah let's just run this script. I'm sure you're down here.


Every New Feature in Python 3.10.0a2

So we have the keys. And that returns us the keys in our dictionary.


Every New Feature in Python 3.10.0a2

And then we also have the values. And that returns a view with the values of our dictionary.


Every New Feature in Python 3.10.0a2

Now we also have items. Which just returns everything inside our dictionary.


Every New Feature in Python 3.10.0a2

So what we have now if we create a keys object here which is our view of the keys in the dictionary.


Every New Feature in Python 3.10.0a2

And let's have a look at the underlying attributes and methods that we have here.


Every New Feature in Python 3.10.0a2

So with this new update with 3.10 we have a new attribute called the mapping attribute.


Every New Feature in Python 3.10.0a2

And you can see it's near the bottom here so it returns the dictionary that this view refers to.


Every New Feature in Python 3.10.0a2

So if we do keys dot mapping it returns us this mapping proxy object which wraps around our original dictionary.


Every New Feature in Python 3.10.0a2

So yeah that's the new the new mapping attribute that we have for our views now.


Every New Feature in Python 3.10.0a2

And with that that is everything that is new to current alpha version of 3.10.


Every New Feature in Python 3.10.0a2


How to build next-level Q&A with OpenAI

Today I want to show you a demo of how you can use generative question answering using Pinecone and OpenAI.


How to build next-level Q&A with OpenAI

Now recently we hosted a workshop between OpenAI and Pinecone and people had the opportunity to ask questions themselves as well.


How to build next-level Q&A with OpenAI

And also if you would like to ask questions you can do use go to this web address up here.


How to build next-level Q&A with OpenAI

And they asked some really good questions.


How to build next-level Q&A with OpenAI

So one of those that I quite liked was how do I define a PyTorch tensor of only zeros?


How to build next-level Q&A with OpenAI

So we come down here we have style at the moment so it's paragraph by question.


How to build next-level Q&A with OpenAI

We'll leave it at that for now.


How to build next-level Q&A with OpenAI

I've just here removed the filters for Streamlit, Huggy Face and TensorFlow.


How to build next-level Q&A with OpenAI

So we're only pulling information from the PyTorch forums but we can add those back in and it should be able to figure out anyway.


How to build next-level Q&A with OpenAI

But when we know we're looking at PyTorch why not add that in there.


How to build next-level Q&A with OpenAI

And then we get this.


How to build next-level Q&A with OpenAI

So there are a few ways to create a PyTorch tensor of all zeros. I don't know why it's re-running there.


How to build next-level Q&A with OpenAI

Oh okay so we get a slightly different answer. Interesting.


How to build next-level Q&A with OpenAI

One way is to use PyTorch function torch zeros.


How to build next-level Q&A with OpenAI

Okay yeah that's cool.


How to build next-level Q&A with OpenAI

And then another one is great using the PyTorch function zeros like.


How to build next-level Q&A with OpenAI

And then it explains what it does as well.


How to build next-level Q&A with OpenAI

So that's quite cool.


How to build next-level Q&A with OpenAI

So it's giving you two options here and also describing how they vary which is really nice.


How to build next-level Q&A with OpenAI

So we can also have a look at the sources.


How to build next-level Q&A with OpenAI

So if we look at the sources we'll see that they're all just coming from PyTorch now.


How to build next-level Q&A with OpenAI

Let's ask another question.


How to build next-level Q&A with OpenAI

So this one's not even really a question.


How to build next-level Q&A with OpenAI

I'm just going to say OpenAI Clip.


How to build next-level Q&A with OpenAI

And what I want to do is just say okay can you summarize what OpenAI Clip is.


How to build next-level Q&A with OpenAI

So we'll come down here. Let's see what it returns.


How to build next-level Q&A with OpenAI

Cool so OpenAI Clip is a contrastive language image pre-training model


How to build next-level Q&A with OpenAI

that uses pairs of images and text and returns a matrix of cosine similarity between text and each image.


How to build next-level Q&A with OpenAI

Okay that's cool. So written in PyTorch uses bcelas.


How to build next-level Q&A with OpenAI

Yeah that's really cool.


How to build next-level Q&A with OpenAI

So that's just a paragraph and then we can also summarize the problems that people have.


How to build next-level Q&A with OpenAI

So essentially this is going to summarize the problems that it sees in the context.


How to build next-level Q&A with OpenAI

So when people are asking questions it's going to summarize those problems that people are having


How to build next-level Q&A with OpenAI

given up to the topic of OpenAI Clip.


How to build next-level Q&A with OpenAI

So we'll come down. It just says this page originally published on that site.


How to build next-level Q&A with OpenAI

Can I rerun see if it comes up.


How to build next-level Q&A with OpenAI

Anything.


How to build next-level Q&A with OpenAI

Actually so we're just going from PyTorch.


How to build next-level Q&A with OpenAI

It's probably not a good idea.


How to build next-level Q&A with OpenAI

We should probably include Hugging Face in there and maybe TensorFlow as well.


How to build next-level Q&A with OpenAI

Okay.


How to build next-level Q&A with OpenAI

So how to use Clip model for image search and style transfer.


How to build next-level Q&A with OpenAI

How to fine tune the Clip model. How to train the model on medical data.


How to build next-level Q&A with OpenAI

And the questions generally seek to find out whether the Clip model can be used to generate replies to English language text input.


How to build next-level Q&A with OpenAI

So that's really cool.


How to build next-level Q&A with OpenAI

So you can see where it's coming from here. So AI. OpenAI's Clip image search, style transfer and so on.


How to build next-level Q&A with OpenAI

So yeah that's I think a really cool example of OpenAI's embedding and generation models used in unison with Pinecone as well.


How to build next-level Q&A with OpenAI

So I want to just show you a little bit of how that works.


How to build next-level Q&A with OpenAI

So over on the left here we have the indexing data set stage.


How to build next-level Q&A with OpenAI

So this starts with scraping data from different websites.


How to build next-level Q&A with OpenAI

So we use the forum websites PyTorch for example here.


How to build next-level Q&A with OpenAI

And also all the ones that you saw here.


How to build next-level Q&A with OpenAI

So Streamlit, PyTorch, Hugging Face and TensorFlow.


How to build next-level Q&A with OpenAI

So I scraped all of those and that resulted in these four JSON line files.


How to build next-level Q&A with OpenAI

Now can I let me show you what they look like.


How to build next-level Q&A with OpenAI

And we just have these lines of all this information. We have the thread.


How to build next-level Q&A with OpenAI

So we have the docs that's coming from the category, which is vision in this case.


How to build next-level Q&A with OpenAI

The thread that's coming from. So like the topic of the conversation in the PyTorch forums.


How to build next-level Q&A with OpenAI

The actual link to it, the question and then we have a load of answers basically or just responses to the question.


How to build next-level Q&A with OpenAI

So we have those and this is what you end up seeing here.


How to build next-level Q&A with OpenAI

So here I've just concatenated them all together.


How to build next-level Q&A with OpenAI

Then we return over to here and we feed all those into an OpenAI embedding model.


How to build next-level Q&A with OpenAI

OK, so if we just went over here to the API.


How to build next-level Q&A with OpenAI

Go to docs and we can come down if we come to embeddings.


How to build next-level Q&A with OpenAI

This talks to us about the embeddings and how they work and how to actually use them.


How to build next-level Q&A with OpenAI

So, yeah, I mean, if you want to have a look at that, you can do it.


How to build next-level Q&A with OpenAI

We're going to see this in a moment anyway.


How to build next-level Q&A with OpenAI

So switching back over to code if we come down a little bit.


How to build next-level Q&A with OpenAI

So here I'm cleaning it up. One thing you can do is that the tokenizer for GPT-3, the embedding model.


How to build next-level Q&A with OpenAI

Is similar to GPT-2 tokenizer.


How to build next-level Q&A with OpenAI

So we can actually download the tokenizer from transformers and check roughly how many tokens we're going to have in our feed.


How to build next-level Q&A with OpenAI

We're feeding it through to the OpenAI API.


How to build next-level Q&A with OpenAI

So go down and then here we can go ahead and actually get some embeddings.


How to build next-level Q&A with OpenAI

So all we're doing here, we get the get embeddings endpoint, which is from.


How to build next-level Q&A with OpenAI

I just write this in. It's just a pip install OpenAI.


How to build next-level Q&A with OpenAI

And you do need an account area everything for this.


How to build next-level Q&A with OpenAI

So I'm just putting that out. You also need a API or API key here.


How to build next-level Q&A with OpenAI

So, yeah, if you need that as well, you go there and then you can see everything.


How to build next-level Q&A with OpenAI

So this is just our order date from before.


How to build next-level Q&A with OpenAI

And then we also have our embeddings that we just created using the endpoint.


How to build next-level Q&A with OpenAI

And I'm just saving them to this query embeddings pocket file.


How to build next-level Q&A with OpenAI

Now at that point, we move over to the index initialization step.


How to build next-level Q&A with OpenAI

So over in our visual that looks like this.


How to build next-level Q&A with OpenAI

So we just take all those, we've got all our text, come through here, created the embeddings.


How to build next-level Q&A with OpenAI

Come over here and now we're at this stage.


How to build next-level Q&A with OpenAI

So the pinecone vector DB, we're just putting all of our embeddings into that.


How to build next-level Q&A with OpenAI

So let's have a look at how we do that.


How to build next-level Q&A with OpenAI

So we first load our data, check that is actually there, looks OK.


How to build next-level Q&A with OpenAI

So the current, this is changing pretty soon.


How to build next-level Q&A with OpenAI

So maybe by the time you're watching this, it might have already changed.


How to build next-level Q&A with OpenAI

But the max size limit for metadata in pinecone is five kilobytes.


How to build next-level Q&A with OpenAI

So what we need to do is just check, OK, is the text field that we're feeding in here greater than that?


How to build next-level Q&A with OpenAI

If so, we need to kind of clean up a little bit.


How to build next-level Q&A with OpenAI

And yes, it is.


How to build next-level Q&A with OpenAI

So in this case, a lot of them are too big.


How to build next-level Q&A with OpenAI

So what we can do is either trim down, like truncate those text fields, or we can just keep the text locally.


How to build next-level Q&A with OpenAI

And then just map the IDs to that text when we're feeding them back in.


How to build next-level Q&A with OpenAI

Now, in this case, I'm just going to map them to IDs.


How to build next-level Q&A with OpenAI

So you imagine we're not going to put our text data in pinecone, just the vectors, but we also include an ID.


How to build next-level Q&A with OpenAI

But obviously, when we're returning the most relevant vectors back to us,


How to build next-level Q&A with OpenAI

we need a way to understand what that text is so that we can feed it into our OpenAI GPT or generation model.


How to build next-level Q&A with OpenAI

So we need a way to go from those IDs to the original text, which is all we're doing here.


How to build next-level Q&A with OpenAI

So I'm getting the IDs, I'm creating an ID, which is literally just numbers, nothing special.


How to build next-level Q&A with OpenAI

It doesn't have to be anything special. It just needs to be unique.


How to build next-level Q&A with OpenAI

And then after we've created those IDs, I'm just initializing the index.


How to build next-level Q&A with OpenAI

You can do this before or after if you want. It doesn't really matter.


How to build next-level Q&A with OpenAI

But we do need to make sure that we're using the cosine metric and also using the correct dimensionality.


How to build next-level Q&A with OpenAI

So we have our embeddings that we've created up here.


How to build next-level Q&A with OpenAI

And here we're just saying, OK, use the dimensionality that is of those embeddings.


How to build next-level Q&A with OpenAI

So I think I'm not sure how much is for the Curie model.


How to build next-level Q&A with OpenAI

It might be something like 2048 dimensionality vectors. I'm not sure.


How to build next-level Q&A with OpenAI

Actually, we can help you look.


How to build next-level Q&A with OpenAI

Go to the Pinecone console, OpenAI.


How to build next-level Q&A with OpenAI

OK, so it's 4096 dimensions.


How to build next-level Q&A with OpenAI

Cool. So we do that and then we just connect to our index after we've created it here.


How to build next-level Q&A with OpenAI

OK, so we create our index.


How to build next-level Q&A with OpenAI

At that point, we can then begin to populate the index with the embeddings that we've created from OpenAI.


How to build next-level Q&A with OpenAI

So that's all I'm doing here. We're doing in batches of 32.


How to build next-level Q&A with OpenAI

So what we're doing is including the ID and then the embedding.


How to build next-level Q&A with OpenAI

And then we're also including metadata. So this is pretty important if you want to do any filtering or, you know,


How to build next-level Q&A with OpenAI

we can see here we've got docs, category and thread and also the link.


How to build next-level Q&A with OpenAI

That's really important for our app because we're including all this information in the sources section that we saw before.


How to build next-level Q&A with OpenAI

We have a look here. We can see we have these sources.


How to build next-level Q&A with OpenAI

So we have the docs followed by the category, followed by the name of the thread.


How to build next-level Q&A with OpenAI

And also if we click on it, here, it will take us through to the thread.


How to build next-level Q&A with OpenAI

So all of that metadata that we specified there is used to build that sources section there.


How to build next-level Q&A with OpenAI

And then finally, so I mentioned before, Pinecone isn't going to at this point handle all of that text information.


How to build next-level Q&A with OpenAI

So all I do is create another dataset which contains the ID to text mappings.


How to build next-level Q&A with OpenAI

And then just save that to file in a JSON file. Nothing crazy there.


How to build next-level Q&A with OpenAI

OK. So that that we just went through was this bit here, which seems like very little for what we just spoke about.


How to build next-level Q&A with OpenAI

But yeah, that bit there. So we're now kind of done with the indexing part. So we can like cross off.


How to build next-level Q&A with OpenAI

We're fine. We've indexed everything. We only need to repeat this again if we're adding more data to our tool.


How to build next-level Q&A with OpenAI

And then we go over to the querying bit. So here we're going to query. This is what the users are going to be doing.


How to build next-level Q&A with OpenAI

So we enter this search term, how to use Radiant Tape Intensifier, as I mentioned before.


How to build next-level Q&A with OpenAI

And if we were going to go through that, it would look like this.


How to build next-level Q&A with OpenAI

OK, so we have the mappings that we have here. So that's the ID to text. We connect to OpenAI.


How to build next-level Q&A with OpenAI

And then we use this get embedding to actually create our embeddings. We don't run that here, though.


How to build next-level Q&A with OpenAI

We load our Pinecone index over here.


How to build next-level Q&A with OpenAI

And then we just define a function that we're going to use, OpenAI, to create the query embedding.


How to build next-level Q&A with OpenAI

And then we use that query embedding to retrieve the most relevant context from Pinecone.


How to build next-level Q&A with OpenAI

And then we feed them back into the generation model and retrieve an answer.


How to build next-level Q&A with OpenAI

OK, so get the embedding. Where is that? So here you can see. So this size here is a query model.


How to build next-level Q&A with OpenAI

You can see that there. And then we're querying Pinecone here.


How to build next-level Q&A with OpenAI

And then we're just going through all of those that we've got and we're creating a really big string of context, which you can see here, actually.


How to build next-level Q&A with OpenAI

So you see we have a create context. How do I use a gradient tape, intense flow?


How to build next-level Q&A with OpenAI

And this creates the context, but not this isn't the answer or this isn't the question that we're going to ask the next generative model.


How to build next-level Q&A with OpenAI

This is just all of the source context that we're going to assign to it. So we're going to say, OK, based on all this information here, answer a particular question.


How to build next-level Q&A with OpenAI

So. Let's go down and we can see, OK, here are the sort of questions we're going to ask.


How to build next-level Q&A with OpenAI

So conservative Q&A. Answer the question based on the context below.


How to build next-level Q&A with OpenAI

And if the question can't be answered based on the context, say I don't know. OK, and go on.


How to build next-level Q&A with OpenAI

So I just want to show you what that might look like in the actual demo.


How to build next-level Q&A with OpenAI

So let's go with, let's restrict everything to streamlit and we'll ask about OpenAI Clip.


How to build next-level Q&A with OpenAI

Maybe. Let's do that. And then we just change the conservative Q&A.


How to build next-level Q&A with OpenAI

And go down and we see that we get I don't know. Right. So that's how the generative model is kind of reading the questions.


How to build next-level Q&A with OpenAI

It's literally reading them and kind of producing this pretty intelligent answer based on what we've given it. It's following the instructions really well.


How to build next-level Q&A with OpenAI

So it had this I don't know. And the reason it didn't know is because what it was given there was not found within the context.


How to build next-level Q&A with OpenAI

OK, there was nothing in the streamlit docs that we filtered down to that said anything about OpenAI Clip.


How to build next-level Q&A with OpenAI

And then as well as that, we also pass in the answer and then we prompted OpenAI's generative model to actually answer that by saying, OK, answer.


How to build next-level Q&A with OpenAI

So it was saying like basically finish this sentence. That's all I'm really doing here.


How to build next-level Q&A with OpenAI

But you get these really incredible responses.


How to build next-level Q&A with OpenAI

OK, and then there's just a few examples here actually. So you see what GPT2, strengths and weaknesses and says I don't know.


How to build next-level Q&A with OpenAI

So it's obviously not enough in there for it to do anything. There's a really good one here, which is the extract key libraries and tools.


How to build next-level Q&A with OpenAI

So embedding models which embed images and text. OK, let me copy this and.


How to build next-level Q&A with OpenAI

And what was it, was it bullet points? Oh no, extract key libraries and tools. Let's do that.


How to build next-level Q&A with OpenAI

And let's make sure we have everything in there. OK, and we get this really cool sort of list of all the embedding models that you would use for extracting or which can embed images and text.


How to build next-level Q&A with OpenAI

So that's really, really impressive. So that is I think pretty much all of that.


How to build next-level Q&A with OpenAI

There's a little more code, I think we go up a little bit. So I kind of skipped over this.


How to build next-level Q&A with OpenAI

So we're just creating a context here, which is what we covered above.


How to build next-level Q&A with OpenAI

And then we're just saying, OK, go into the OpenAI completion endpoint.


How to build next-level Q&A with OpenAI

And that's going to complete the question or the text that we've given to it. So that's really cool.


How to build next-level Q&A with OpenAI

And yeah, no, this sort of thing just almost blows me away a little bit with how impressive it is because it seems almost genuinely intelligent.


How to build next-level Q&A with OpenAI

And you ask a question that gives you like a genuinely intelligent response that is kind of pulled from all these different sources and formatted in a particular way based on what you've asked.


How to build next-level Q&A with OpenAI

It is for me incredibly interesting and definitely something I want to read about more.


How to build next-level Q&A with OpenAI

So generative models because it's clearly really impressive and very useful.


How to build next-level Q&A with OpenAI

So that's it for this video. I hope this has been interesting to see this demo and sort of go through how everything works.


How to build next-level Q&A with OpenAI

But yeah, let me let me know what you think.


How to build next-level Q&A with OpenAI

And if you are interested in trying out, you can do use to share streamlet IO, Panconi playground, beyond search, OpenAI and so on.


How to build next-level Q&A with OpenAI

And if you'd like to take a look at the code for that as well, you can do.


How to build next-level Q&A with OpenAI

What you do is head over to this URL here and then you have all the code I just showed you in here and also the data as well.


How to build next-level Q&A with OpenAI

If you if you'd like that. So, yeah, I think it's been really cool. So thank you very much for watching.


How to build next-level Q&A with OpenAI


Evaluation Measures for Search and Recommender Systems

Evaluation measures are the foundation behind many of the biggest tech companies in the world.


Many of these companies have publicly stated that a big part of their engagement is thanks to well-implemented information retrieval systems.


All of these systems rely on retrieving relevant information from some sort of database.


These evaluation measures are how we can understand whether our retrieval performance is good or not.


Evaluation measures are split into online and offline metrics.


Online metrics can only be measured with a deployed product.


Usually this begins with A-B testing.


But before we even reach the stage of A-B testing, there is another step.


Another set of evaluation measures that we call offline metrics.


These are critical because they help us predict our retrieval performance before needing to put it in front of users.


Now an organization should always use both online and offline metrics.


But offline metrics are the starting point and that's what we're going to focus on in this video.


We are going to have a look at predicting the performance of an information retrieval system before deployment, before A-B testing, before any of that.


Now there are many different types of offline metrics but some of the most popular that we are going to focus on are recall at K, mean reciprocal rank,


the mean average precision at K and the normalized discounted cumulative gain at K.


All of these metrics at first glance might seem complicated but as you will see they're very simple and incredibly useful ways of measuring the performance of our systems.


Now within offline metrics including those I just described, we have a separation into order unaware and order aware.


This refers to whether the order of the results has any impact on the final score.


In order aware, the order of results doesn't really make a difference whereas with order aware they do.


Now we'll see a few examples of this throughout the video.


So if it doesn't make sense yet don't worry about it we will cover all of this.


Throughout this video we are going to be using a very small dataset of just eight images.


Now in reality you will typically be working with millions or even billions of items.


So this is a very simplified version but it's ideal for understanding how each one of these metrics works.


So in this case our query is cat and a box and we may return any eight of these images in any particular order.


Of course some of them are relevant and some of them are not.


And we can see those relevant results here so the actual relevant results refer to those results that we would like to return.


And everything else is not relevant to us so we would rather not return those results.


Now when evaluating the performance of our information retrieval system we are going to be comparing actual conditions to predicted conditions.


Now an actual condition refers to the true relevance of an item in the dataset.


So our actual relevant items here.


Their actual condition is positive because they are relevant.


And for the other ones that are not relevant their actual condition is negative.


And we want to get our predicted conditions close as possible to these actual conditions.


Predicted conditions are what is produced by our information retrieval system.


Now the information retrieval system may predict something like this.


So say we ask the information retrieval system to return two items.


It may rank everything and the top two items I believe to be the most relevant to our particular query kind of box are here.


We have number one which has a predicted condition of true which is what you can see over here with this P.


OK so that P hat that you see means that the predicted condition is true.


But we can also see that there's a negative.


OK and that is because the actual condition of that item is false.


It is not relevant to us. OK so we get this false positive.


So it's negative actual negative and predicted positive.


On the other hand we have this other returned item and this is a good result because we have a predicted positive.


And an actual positive so that's a true positive.


And we can also see down here we have these other items so again another good result.


This is a true negative because the actual condition is negative and so it's a predicted condition.


And if we go a little further we have some negative results although you can't really see the box here.


You can just see in the image there.


This is a false negative because it is actually in a box.


OK and we're predicting it or the information retrieval system has predicted this is not relevant when it is actually relevant.


So just remember we have the actual conditions predicted conditions and we have positives or negatives based on those two forms of conditions.


Now with that we can move on to our first metric which is recall at K.


Recall at K is pretty simple. We have the true positives which we just described divided by the true positives and false negatives.


So what does that mean? We take a look at the predicted positives that are also actual positives.


So the good positively return results.


OK that's what this PP is and we divide that by the total number of actual positives in the entire data set whether it was returned or not.


OK so here they were returned here they were not. That's all the recall is. It's relatively simple.


So let's have a look at this. We have three items that should have been positive in this retrieval process but they were not retrieved.


They are the false negatives. So we add them to the denominator of our recall formula and then we have one true positive.


So one that was an actual positive result and it was also predicted positive which is good but it could have been better of course.


So we add that to the denominator as well and also to the numerator. So we get 1 over 4.


So 0.25 for our recall at 2. Now it's a little bit unfair because even if this was a perfect retrieval system there are four actual positives here.


I'm only returning two items so the best it can score is 0.5. But that is just how the recall metric works.


Now let's have a look at how we would calculate that in Python. So we just create a recall function here.


All we're doing is taking the actual items the predicted items up to the top k.


So before we had recall at 2. So in this case k would be 2 and we simply do that calculation that you saw before.


Nothing complicated there. That's our recall function and all I'm doing here is saying these are the actual results


and these are up to the max of what we could predict.


Because what I'm doing here is going through and calculating this for all of the possible k values up to the number of items we have in the dataset.


Just so you can see how recall works. So at the lower k values recall scores a lot lower than does at higher k values.


Now if you have a pretty low k value and your system is still performing well that's pretty good news.


But if you increase your k value a lot you can kind of cheat the system because here we're getting a perfect recall score


and we know that our performance is not perfect and that's obviously not ideal.


And another problem with recall is we could swap in here with a spot item 1 and 2.


So we could be predicting number 2 as our top result and recall at least recall at 2 would still return us the same score.


And depending on your use case this may or may not be a problem.


But if it's important for you to retrieve the most relevant results at the top of your retrieval list then this can be a problem.


If it doesn't really matter if you're just returning let's say the 10 items and you just need one item to be the one relevant item to be within that 10


you don't really care where it is you can go ahead and order unaware metrics like recall work perfectly fine.


But if you would like one item to be at score at rank 1 then these order unaware metrics become a problem.


So moving on to order aware metrics we will start with the mean and reciprocal rank or MRR.


MRR has a few advantages over recall.


First it considers the order of our return results as you will see that in a moment.


And it also considers multiple queries so we're not just relying on a single query in the case of MRR.


So if we take a look at the formula here we there are a few things to consider here.


First we have the reciprocal rank which is what you can see over here.


Then we sum the reciprocal rank for each query Q from Q1 to the total number of queries which is a capital Q.


And then we take the mean by dividing this summed value by the number of queries that we made.


That's how we get MRR but we'll go through and break it down a little further in an example.


So we start with these three queries.


We have cat in a box, white cat and dark cat.


So queries two and three similar but slightly different to query one.


Now the data set is the same so we're still returning all of the same items we're returning them in a different order.


So we first need to calculate the rank reciprocal.


The rank reciprocal is the one over rank Q that you saw before.


Now what this does is returns the rank of the first relevant item in the rank Q position.


That's what rank Q means.


So for your query Q it returns the first relevant rank.


So in an ideal scenario this would be one.


So if we take a look at query one over here the first item first relevant item is in rank two.


So that means here that rank one for query one is one over two.


OK so we get 0.5.


Over here we get the ideal scenario where it is returning a relevant item at position or rank position one.


So we get one over one.


And then over here we get a worse result.


So we have to go all the way down to rank five to find the first relevant item.


And that means here we have to do one divided by five.


So that's what the rank reciprocal part of MR is.


And then we need to sum all those together.


OK so we have all those and that leaves us with 1.7.


OK but that's not everything we still need to divide by the total number of queries.


So we have 1.7 divided by Q which is three because we made three queries which is equal to 0.57.


Now how does that look in Python?


Again it's pretty simple we're just going to replicate what we just did.


So we have the actual relevant results.


OK we would run those.


Then we come down to our calculation.


We have a total of three queries here.


So our Q value the number of queries is three.


And then we're just going to look through each of those queries.


We're going to make our query and we're going to see what the first relevant result is.


OK so the first relevant result is going to be two one and five.


OK and we see that down here.


And then all we do is calculate the average reciprocal based on the number of queries we made.


Which is 0.57 which is the same as what we got before.


And now we can move on to the mean average precision.


Now mean average precision or MAP is another popular order where metric has a bit of an odd name.


It's a mean of the average precision.


So the mean of an average but it does make sense as we'll see in a moment.


So we're going to start by just calculating the precision.


OK so very similar to the recall but this time only considering items that were actually returned.


OK so the items that were actually retrieved.


And you can see that because we're just returning the predicted positives.


OK and the denominator here is basically just K.


OK so we can actually just do this.


And that gets us our precision at K.


So the number of relevant return results divided by the total number of return results.


Now let's return to our kind of box example.


In this case we are basically just adding one one because we've returned two items here.


So it's everything we returned e.g. K.


And we've returned one relevant item such as 1 over 2.


So we replace this with K and then 1 over 2.


It's pretty simple. That's precision.


And precision is the first component of mean average precision.


But before we actually get to MAP at K we need to go to average precision at K.


Now to calculate that we take the precision value which we just calculated.


But you'll notice that we're using a small K here.


And that's because we're going through every K value from 1.


So precision at 1 all the way up to the actual K value that we have provided.


So in our data set we go all the way up to 8.


And we would calculate the precision at 1, 2, 3 and so on all the way up to 8.


And what this is doing here. This is called a relevance parameter.


And it's looking at the relevance of a item at position K.


So what that means is here we have our precision values that we have calculated.


This here is a low case K by the way not capital.


And the relevance score for each one of these if it is not relevant is 0.


And if it is relevant is 1.


And we go through all of those taking the zeros for not relevant as you can see here.


And the ones as relevant. And we multiply all those together.


So essentially what we do there is just cancel out all the non-relevant calculations.


And only calculate the precision where we have a relevant item.


Which is what you get here.


So in the end what we end up doing is we take all those values.


So you see 0.5, 0.5, 0.6, 0.57 which is the precision values that we get here.


And we just divide them by the number of relevant results that we've returned.


Which is 4. And that is our average precision values. Average over the possible K values.


And you can see here that we've done this for three queries.


So we have query 1, 2 and 3.


Now the mean part of mean average precision is simply taking the average over each one of these.


And that's mean average precision.


So how do we implement that in Python?


We're going to use the same actual relevant items this time.


Again we're using Q like we did before. Number of queries and we have our predicted.


It's just the same for each one of our queries in this toy example.


We just go through and we calculate the precision first.


Which is pretty straightforward.


We check the relevant parameter values.


It's either 1 or 0 depending on whether the item at position K is relevant or not.


And then we calculate the numerator value for AP.


So the average precision.


And simply divide that by the number of actual items for that particular query.


And then we get these numbers here.


And then all we need to do is take the mean of all those AP values to get map at K.


Which we have here.


It's relatively straightforward.


Now the final metric I want to talk about here is a very popular one and incredibly useful.


It basically takes everything that I've just said about all the other metrics we've used and puts them all together.


This is called the Normalized Discount Cumulative Gain at K metric.


It's another order aware metric.


And we derive it from a few simpler metrics.


So we start with the Cumulative Gain at K metric.


So Cumulative Gain uses again this relevance parameter.


But the relevance parameter is slightly different this time.


It's not just 0 or 1 binary values.


Instead it is a ranking that is assigned to every single item for every single query.


And they are assigned from a scale.


The scale can vary.


We're going to use 0 to 4 in this example.


They are assigned a value of 0 which is less relevant.


Up to 4 which is the most relevant that an item can be for a particular query.


Now of course this means that your data has to be labeled well.


Which is a disadvantage of using this metric.


But if you have that labeled data it's ideal because you have a lot more expressibility with your evaluation metric.


So let's have a look at an example of how that might work.


So going back to our earlier example, change it slightly.


Now we're looking for a white cat in a box.


We have two items that are super relevant because they are white cats in boxes.


We have number 2 over here and number 5.


They are both white cats in boxes. They are perfect.


So the relevance score for those we've assigned both of them a 4.


Which is the most relevant an item can be for that particular query.


And then we have some other things. So let's go down to 3s.


So here at number 4 we have a cat in a box but it's not white.


And the same for number 7.


So it's still relevant that there's a cat in a box but it's not a white cat.


So it's less relevant.


And go down to number 2.


So here item number 8 and it's being ranked 2 because it's a dog in a box.


So it's a domestic animal in a box.


So it's sort of in the middle. It's kind of relevant but it's not relevant at the same time.


And then we get further away from that.


So we go down to rank 1.


So we have item 3 and item 6.


Item 3 is just a box.


Item 6 it's 2 cats.


So it's kind of half of what we're looking for.


So in this case it's less relevant.


But neither of those are as irrelevant as a dog.


In this case the dog has been ranked as having a relevant score or rank of 0.


Because it's not a cat, it's not a box, it's not being really too likely other than being a domestic animal.


Which is not what we're looking for.


So that's how that scoring mechanism works with all of the cumulative gain metrics that we're going to look at.


And if we were to take that into our formula.


We would get something like this.


So we have...


We're summing over all k values.


So we have k here.


Up to the k of 2.


So this is the lower case k, this is the upper case k.


So we go 1 and 2.


So we have a look at number 1.


It's not relevant.


It has a score of 0.


So it's equal to 0.


And then number 2 is relevant.


So it has a score of 4.


So our cumulative gain score in this case is equal to 4.


Now it's important to note here that NDCG, the metric, is order-aware but CG by itself is not.


If we've swapped those two items as we have here.


So we've swapped these two.


If we go back to this formula.


We are now doing 4 plus 0.


Which again is just equal to 4.


So cumulative gain by itself is not order-aware.


To make it order-aware, we have to use a discounted cumulative gain or DCG metric.


DCG adds one more component which is this penalty value in the denominator.


Using this penalty function.


So this should be 4 and this should be 4 and 0.


So we take a look at these.


We get 0 plus 2.52.


So we score 2.52 in the case of the first query where we have the not good result in position in rank 1.


Whereas if we have a good result in rank 1.


We get a score of 4 because log to the base 2 by 2 is equal to 1 plus 0.


OK so we get a score of 4 for our second query which is better.


And that's great because we're returning a better score.


Now let's have a quick look at how we can calculate the DCG in Python.


So we start by importing log to the base 2 from the math library.


We have the same relevant scores that you saw before.


So that's the relic k values.


And we just go through those and we calculate the cumulative or discounted cumulative gain using relic k and our penalty function there.


OK you can see as it increases because it's cumulative.


The DCG also increases.


Now one problem with DCG as we've just seen is that the values don't really have any range because it depends on the relevance rankings or the range of relevance scores that we've assigned.


In this case we use 0 to 4.


But you could use 0 to 100 if you want to be excessive.


But you could do that.


And in that case the values that you're going to get out of this are going to be completely different than if you use a range of 0 to 4 which makes interpreting these numbers very difficult.


So that's where the normalized DCG comes in.


So the normalized DCG or NDCG uses another DCG metric called the ideal DCG to normalize these values to within a range of 0 to 1 where 1 is basically perfect and 0 is terrible.


Now to calculate the IDCG all we need to do is reorder all of our ranked results into the ideal order.


So those results that have the highest relevance will be ranked first and it would go through in terms of relevance score.


And then we just calculate DCG again using that new order because that is the ideal DCG.


So using our earlier example we have two items that are both ranked at 4.


So we cross these values out and in this case we're going to use 4 instead.


And we would get a value of 4 plus 2.52.


So the ideal DCG is equal to 6.52.


Now to calculate that in Python it's incredibly simple.


All we do is sort all of our relevance ranks beforehand using var equals true and do the exact same thing calculate DCG.


And using that we see we get these values.


So IDCG at 2 is equal to 6.52.


And with all those calculations done we have our DCG and our IDCG and now we can calculate the normalized DCG.


Which is simply DCG divided by the ideal DCG.


So for our DCG value we got using query 1 of 2.52.


And our IDCG is 6.52.


And that leaves us with a not too great score of 0.39.


Now switch back over to Python again and see how we would do that for all of our k values up to 8.


So running through that all I'm doing is calculating the ideal values here and also the normal discounted or DCG values.


And dividing DCG by IDCG and then we get these values.


So you can see over time as we include more items in there the score does improve.


But it never gets to perfect because it's not perfect or ideal ranking of all of our items.


Now NDCG is a great offline metric because it really prioritizes for highly relevant documents.


But at the same time we do need that more complex training data or label data in order for this metric to actually work.


But if we do have that data this metric is perfect.


Now that's the end of our metrics.


That's all of them.


So we've covered recall at k, we've covered mean reciprocal rank, we have covered mean average precision and we have also covered NDCG.


You can use a few of those in your information retrieval systems to evaluate your performance.


And you really don't need much else other than that for your offline metrics.


In the case of Spotify for example with their podcast search they use mean reciprocal rank at 30 and also recall at 30.


And they were there on offline metrics.


They didn't use anything else other than recall at 1 for their in batch evaluation.


Now of course these metrics still need to be supported with online metrics during A-B testing and whilst you're actually in production.


But these really get that first step of preparing an information retrieval system that is the best it possibly can be without needing to deploy it to any users.


And that's ideal because you will get the best information retrieval system you can if you use these offline metrics well.


So that's it for this video.


I hope this has been useful.


Spotify's Podcast Search Explained

In the past few years, podcasts have become increasingly popular


Spotify's Podcast Search Explained

and leading the charge in podcasts is Spotify.


Spotify's Podcast Search Explained

Now, Spotify only very recently entered the market for podcasts in 2018.


Spotify's Podcast Search Explained

Despite that, in a few short years, so by 2021,


Spotify's Podcast Search Explained

they had already usurped Apple as the leader in terms of monthly active listeners.


Spotify's Podcast Search Explained

Now, Apple has been doing this for a very long time, since I think it's 2008 or maybe even earlier.


Spotify's Podcast Search Explained

But despite that, Spotify are now ahead of everyone.


Spotify's Podcast Search Explained

And of course, to back their sort of investments in podcasts,


Spotify's Podcast Search Explained

they've also invested heavily in the technology that powers different components of the podcast experience.


Spotify's Podcast Search Explained

Whether that's Anchor, which is an all-in-one app that allows people to record and publish podcasts,


Spotify's Podcast Search Explained

or what we're going to focus on, a natural language search for podcasts.


Spotify's Podcast Search Explained

Now, Spotify's natural language search or semantic search for podcasts is, I think, a really interesting use case


Spotify's Podcast Search Explained

because it enables a more intuitive user experience by allowing us to search through their podcast catalog,


Spotify's Podcast Search Explained

which is huge, using more natural language queries, rather than the typical keyword or term matching that you would do in most places.


Spotify's Podcast Search Explained

So beforehand, where we were searching, we have to kind of know the words that we're looking for,


Spotify's Podcast Search Explained

or the terms that we're looking for.


Spotify's Podcast Search Explained

And I think we as humans, we don't really know what exactly we're looking for in terms of the words or keywords


Spotify's Podcast Search Explained

that are in whatever it is we're looking for.


Spotify's Podcast Search Explained

Instead, we tend to think more in concepts and ideas.


Spotify's Podcast Search Explained

Our language is meaningful. It's not specifically about term matching or keyword matching.


Spotify's Podcast Search Explained

So it makes sense that we might want to create a search experience that replicates the meaning in language,


Spotify's Podcast Search Explained

rather than just terms and words.


Spotify's Podcast Search Explained

So imagine we wanted to find a podcast that talks about eating healthily over the winter holidays.


Spotify's Podcast Search Explained

We might search something like this.


Spotify's Podcast Search Explained

We eat better during X-mas holidays.


Spotify's Podcast Search Explained

Now, in the dates that we're going to be using, there is a podcast description that talks about exactly this.


Spotify's Podcast Search Explained

Its description is, Alex Straney chats with Dr. Priya Alexander about how to stay healthy over Christmas


Spotify's Podcast Search Explained

and about her letter to patients.


Spotify's Podcast Search Explained

Now, if you compare those two, they don't share any of the same words,


Spotify's Podcast Search Explained

despite pretty much being about the same thing.


Spotify's Podcast Search Explained

So if you try and use this in a term matching query, it's not going to work.


Spotify's Podcast Search Explained

But if we were to spot that for an actual language query or a semantic search, it does.


Spotify's Podcast Search Explained

Because what we are searching for here, the meaning overlaps.


Spotify's Podcast Search Explained

The genuine human meaning behind those two phrases is very similar.


Spotify's Podcast Search Explained

And when semantic search is done properly, it is able to identify, OK, we're looking for the meaning behind these.


Spotify's Podcast Search Explained

And the meaning of these two does overlap.


Spotify's Podcast Search Explained

We're talking about Christmas holidays, being healthy, eating better.


Spotify's Podcast Search Explained

They're very similar concepts.


Spotify's Podcast Search Explained

But enabling meaningful search in this way is not easy.


Spotify's Podcast Search Explained

So what I want to do is actually have a look at how Spotify have done this


Spotify's Podcast Search Explained

and actually replicate it and see what we can do as well.


Spotify's Podcast Search Explained

So the technology powering Spotify semantic search consists of two different components.


Spotify's Podcast Search Explained

We have the NLP or natural language processing side of it.


Spotify's Podcast Search Explained

And we also have the vector search component.


Spotify's Podcast Search Explained

Now, these technologies can be seen as two steps in the search process.


Spotify's Podcast Search Explained

Given a natural language query, a language model, so this is the NLP side,


Spotify's Podcast Search Explained

will take that query and convert it into something we call a dense vector,


Spotify's Podcast Search Explained

which is essentially a meaning encoded numeric representation of whatever that query is.


Spotify's Podcast Search Explained

OK, sounds confusing, but we'll see that it's not really too complex quite soon.


Spotify's Podcast Search Explained

These dense vectors can then be compared as you would compare normal vectors.


Spotify's Podcast Search Explained

So imagine you have two points in a particular space and you want to calculate the distance between them.


Spotify's Podcast Search Explained

We can literally do that, but with the meaning behind these queries.


Spotify's Podcast Search Explained

So let's have a look at an example of that.


Spotify's Podcast Search Explained

So here we just have a 3D chart.


Spotify's Podcast Search Explained

And typically when you have these dense vectors, they have many dimensions.