Q&A Document Retrieval With DPR

paste them here and


Q&A Document Retrieval With DPR

This would just initialize it from what we've already built so it's using the same index that already exists


Q&A Document Retrieval With DPR

so


Q&A Document Retrieval With DPR

Just initialize that and once we have our document store. Okay, cool. We have that now


Q&A Document Retrieval With DPR

Now what we want to do is set up our DPR, which is a dense passage retriever, which


Q&A Document Retrieval With DPR

essentially uses


Q&A Document Retrieval With DPR

dense vectors and


Q&A Document Retrieval With DPR

a type of efficient similarity search


Q&A Document Retrieval With DPR

to embed these indexes as


Q&A Document Retrieval With DPR

dense vectors and then once it comes to actually searching


Q&A Document Retrieval With DPR

And finding the most similar or the most relevant


Q&A Document Retrieval With DPR

Documents later on it will use those dense vectors and find the most similar ones


Q&A Document Retrieval With DPR

So I'll explain that a little bit better in a moment


Q&A Document Retrieval With DPR

So first what we want to do is actually initialize that


Q&A Document Retrieval With DPR

So we do from Haystack dense retriever


Q&A Document Retrieval With DPR

Import dense passage retriever


Q&A Document Retrieval With DPR

Sorry, it's the other way around here so retriever dense


Q&A Document Retrieval With DPR

And


Q&A Document Retrieval With DPR

then we'll put into a


Q&A Document Retrieval With DPR

Variable called retriever which uses the dense passage retriever from up here


Q&A Document Retrieval With DPR

And in here we need to pass a few parameters. So the first thing is the document store. So the document store is


Q&A Document Retrieval With DPR

just what we've already initialized up so and


Q&A Document Retrieval With DPR

Then we need to initialize two different models so it's the query embedding model


Q&A Document Retrieval With DPR

And the passage embedding model


Q&A Document Retrieval With DPR

Now behind the scenes Haystack is using the Hugging Face Transformers library


Q&A Document Retrieval With DPR

So what we'll do is we'll head over to the


Q&A Document Retrieval With DPR

Models over there and see which embedding models we can use for DPR


Q&A Document Retrieval With DPR

Okay, so here let's just search for DPR and you'll find we have all of these models from Facebook AI


Q&A Document Retrieval With DPR

Now with DPR


Q&A Document Retrieval With DPR

the reason that it's so useful for question answering is that we have


Q&A Document Retrieval With DPR

What are two different models that encode


Q&A Document Retrieval With DPR

the text that we pass into it so we have


Q&A Document Retrieval With DPR

this sort of setup during training and


Q&A Document Retrieval With DPR

What we see down here


Q&A Document Retrieval With DPR

Are these two models we have this


Q&A Document Retrieval With DPR

EP, BERT, and EMP


Q&A Document Retrieval With DPR

models we have this EP, BERT encoder


Q&A Document Retrieval With DPR

And we also have this EQ, BERT encoder. Now the EP, BERT encoder encodes the passages or the context


Q&A Document Retrieval With DPR

So essentially the paragraphs that we have


Q&A Document Retrieval With DPR

fed into our elastic search model


Q&A Document Retrieval With DPR

This is what we'll be


Q&A Document Retrieval With DPR

encoding them into these vectors here


Q&A Document Retrieval With DPR

Now this is during training this whole graph. So all we will actually see


Q&A Document Retrieval With DPR

when we're encoding these vectors is we will see the


Q&A Document Retrieval With DPR

EP encoder


Q&A Document Retrieval With DPR

And this will create the EP vectors


Q&A Document Retrieval With DPR

And all we're going to do is feed in all of the documents


Q&A Document Retrieval With DPR

from elastic search into this


Q&A Document Retrieval With DPR

Now once all of these have been encoded


Q&A Document Retrieval With DPR

We then have a new set of dense vectors


Q&A Document Retrieval With DPR

And then we'll have a new set of dense vectors


Q&A Document Retrieval With DPR

And all of those


Q&A Document Retrieval With DPR

Will be fed back into our document store so back into elastic


Q&A Document Retrieval With DPR

Now when it comes to performing similarity search later on


Q&A Document Retrieval With DPR

We're going to ask a question and that question will be processed by the EQ encoder


Q&A Document Retrieval With DPR

So here we have our


Q&A Document Retrieval With DPR

EQ encoder


Q&A Document Retrieval With DPR

And we have our question so that will go into here


Q&A Document Retrieval With DPR

And that will encode our question


Q&A Document Retrieval With DPR

And then send it over to elastic and say okay what are the most similar vectors to


Q&A Document Retrieval With DPR

this vector that we created from a question


Q&A Document Retrieval With DPR

And the reason that we're asking this question is because


Q&A Document Retrieval With DPR

We're using the same question


Q&A Document Retrieval With DPR

And the reason that DPR is so good is that if you look at the training down here


Q&A Document Retrieval With DPR

We are creating these EP vectors and these EQ vectors that are matching so where we have a matching question to a matching context


Q&A Document Retrieval With DPR

We are training them


Q&A Document Retrieval With DPR

to maximize the dot product


Q&A Document Retrieval With DPR

Because the dot product


Q&A Document Retrieval With DPR

Is between those two vectors so what happens is


Q&A Document Retrieval With DPR

that


Q&A Document Retrieval With DPR

a relevant passage and a relevant question


Q&A Document Retrieval With DPR

Will come out to have a very similar vector


Q&A Document Retrieval With DPR

So one example that I like to use


Q&A Document Retrieval With DPR

Is if our question


Q&A Document Retrieval With DPR

Was


Q&A Document Retrieval With DPR

What


Q&A Document Retrieval With DPR

Is the capital of


Q&A Document Retrieval With DPR

France


Q&A Document Retrieval With DPR

The embedding that I will create from that will create a context that looks something like the capital


Q&A Document Retrieval With DPR

Of France


Q&A Document Retrieval With DPR

Is and you know something here we don't know what it will put because it doesn't actually know what the capital of France is


Q&A Document Retrieval With DPR

It's just doing linguistic


Q&A Document Retrieval With DPR

Transformations to try and figure out what sort of context the answer would come from then of course when you feed this context


Q&A Document Retrieval With DPR

into elastic


Q&A Document Retrieval With DPR

The most similar


Q&A Document Retrieval With DPR

Vector will be the one which contains the answer to our question


Q&A Document Retrieval With DPR

Okay, because the answer to our question which is something like the capital of France is Paris


Q&A Document Retrieval With DPR

Now we don't have Paris here


Q&A Document Retrieval With DPR

But it will be able to figure that out because it will be the most similar sequence to the context that DPR has


Q&A Document Retrieval With DPR

produced


Q&A Document Retrieval With DPR

Now


Q&A Document Retrieval With DPR

back to


Q&A Document Retrieval With DPR

Hugging face here. You can see we have these multiple DPR


Q&A Document Retrieval With DPR

Models and we have a few of them here


Q&A Document Retrieval With DPR

These multiple DPR models and what we want is a pair. We want a question encoder


Q&A Document Retrieval With DPR

And a ctx which is context encoder now, we'll be using this single nq base


Q&A Document Retrieval With DPR

So what I'll do is just copy this


Q&A Document Retrieval With DPR

And in here we just add it in our model


Q&A Document Retrieval With DPR

Okay, so that's a question encoder


Q&A Document Retrieval With DPR

Now what we also need is the context encoder


Q&A Document Retrieval With DPR

Which is instead of question here


Q&A Document Retrieval With DPR

We just add ctx


Q&A Document Retrieval With DPR

Now we have two other parameters that we need to add in here


Q&A Document Retrieval With DPR

Which is use GPU which is if you're using a GPU obviously you set this to true if not you go with false it will


Q&A Document Retrieval With DPR

Take a little bit of time to process this if you're not using a GPU though


Q&A Document Retrieval With DPR

Then we also add embed


Q&A Document Retrieval With DPR

Title equals true as well now


Q&A Document Retrieval With DPR

What we should


Q&A Document Retrieval With DPR

See is this will execute without error, hopefully


Q&A Document Retrieval With DPR

Okay, great. And then what we need to do is update the embeddings within elastix search. So what we've done here is


Q&A Document Retrieval With DPR

Kind of set up the process and now what we need to do is update


Q&A Document Retrieval With DPR

The documents that we have in elastix search to have dpr embeddings. So to do that we go


Q&A Document Retrieval With DPR

doc store


Q&A Document Retrieval With DPR

update embeddings


Q&A Document Retrieval With DPR

And then in here we pass our retriever


Q&A Document Retrieval With DPR

Okay, now this may take well this will be ready for the next step


Q&A Document Retrieval With DPR

May take well, this will be really quick for me. We don't have that many documents


Q&A Document Retrieval With DPR

And even on cpu actually with the lack of documents we have it should be pretty quick


Q&A Document Retrieval With DPR

so


Q&A Document Retrieval With DPR

What we see here we created these embeddings


Q&A Document Retrieval With DPR

And then we posted them again to our


Q&A Document Retrieval With DPR

Index


Q&A Document Retrieval With DPR

So


Q&A Document Retrieval With DPR

That is pretty cool. And now what we need to do is just test that it actually works now


Q&A Document Retrieval With DPR

Let's go with retriever and this is how we


Q&A Document Retrieval With DPR

Get context from our elastix search document store. It's all right retrieve


Q&A Document Retrieval With DPR

And then we pass in a query here so let me just find something here


Q&A Document Retrieval With DPR

Like what did you learn from your


Q&A Document Retrieval With DPR

great grandfather maybe


Q&A Document Retrieval With DPR

or


Q&A Document Retrieval With DPR

from Verus


Q&A Document Retrieval With DPR

Yeah, let's go


Q&A Document Retrieval With DPR

from grandfather. Let's go grandfather


Q&A Document Retrieval With DPR

What did you what did your grandfather teach you?


Q&A Document Retrieval With DPR

Now, I don't know if this is going to work, but let's see


Q&A Document Retrieval With DPR

Okay, so you see that we return quite a few


Q&A Document Retrieval With DPR

Contexts here


Q&A Document Retrieval With DPR

Now we haven't set up the full thing. So we're just returning


Q&A Document Retrieval With DPR

What it sees as being relevant context we are not actually extracting an answer out yet


Q&A Document Retrieval With DPR

Because that will be the job of our reader model


Q&A Document Retrieval With DPR

So what we have is from my great grandfather we have that one so it's okay


Q&A Document Retrieval With DPR

Some other ones here


Q&A Document Retrieval With DPR

Let's type in grandfather


Q&A Document Retrieval With DPR

Okay, so it's just returning that one which is fine it's not perfect but


Q&A Document Retrieval With DPR

What we would expect to do in reality is return more


Q&A Document Retrieval With DPR

So let's try another one as well


Q&A Document Retrieval With DPR

And let's say


Q&A Document Retrieval With DPR

Who taught you


Q&A Document Retrieval With DPR

about freedom of will?


Q&A Document Retrieval With DPR

Who taught the freedom of will?


Q&A Document Retrieval With DPR

And


Q&A Document Retrieval With DPR

We see here. Okay in the first one. We don't get the correct answer that we want or the correct context


Q&A Document Retrieval With DPR

And we go down and I saw there he is. So


Q&A Document Retrieval With DPR

here


Q&A Document Retrieval With DPR

is the


Q&A Document Retrieval With DPR

Context that we wanted to return


Q&A Document Retrieval With DPR

So it returns that as the fourth best context


Q&A Document Retrieval With DPR

Which is fine because when we build our reader model later on


Q&A Document Retrieval With DPR

We kind of expect that to sort those a little bit better than our


Q&A Document Retrieval With DPR

Retriever model


Q&A Document Retrieval With DPR

This is pretty cool and I think definitely a good start. So now what we have


Q&A Document Retrieval With DPR

Retrieved the meditations


Q&A Document Retrieval With DPR

Set up our document store


Q&A Document Retrieval With DPR

And now we have also set up our retriever


Q&A Document Retrieval With DPR

So we can also cross that off


Q&A Document Retrieval With DPR

And next thing is our reader model


Q&A Document Retrieval With DPR

So I think that's it for this video in the next one, of course, we'll move on to that reader model


Q&A Document Retrieval With DPR

And let's just see how that goes


Q&A Document Retrieval With DPR


Hugging Face Datasets #2 - Dataset Builder Scripts

Today we're going to continue with the Huckabase data sets series


And we're going to have a look at how to use the builder scripts


So with the builder scripts we can either we can do a few things so we can


include data pre-processing


within the data loading pipeline


We can stream from a another sort of remote data source, which is pretty useful if you are


using a data set where the owners of that data set want the data to be streamed from their server


Which happens quite a lot or if you know, maybe you have your data set split into multiple


Files or you have images in your data set or something along those lines


In those cases, you always need to use one of these data set building scripts


So what I'm first going to do very quickly is show you how I created a compressed file for this demo


So we're going to create


Let me show you so we're gonna go over here. So into this James Callum HF data sets repo on github


01 builder script we're going to here and you you will see this file here. So this data set tar gz file


So this is a zip or compressed file and we're actually going to stream our data from


This exact location. So if you


We go on here. I see we have this download link download button. We're just going to copy that link address and we're gonna use that to


Stream our data into into the data


data set building script


so very quickly


How did I build that you can actually have a look at this file here?


So all I'm doing is I'm taking this the reddit topics data set that I've built already very similar to the dates that we used


in the last video


It's just a little bit bigger. So it's not massive


3.8 thousand rows


All it converts pandas convert to a dictionary or using the the records orientation


and then


Save that as a JSON L or JSON lines file


then


we


compressed it using this so you'll probably if you have your own data set and you want to compress it and


Kind of follow the same steps of doing here. This is what you will need. So you need to add


your data set file to the


Zipped compressed file here and you choose to half file that this I believe is actually installed by default


With Python so you won't have to pivot install that so with all of that


We can go ahead and actually look at how we build our data building script. So we start with a template


first so come over to


hugging face here and go data sets and


And let's go squad, okay, so


go here so squad is just a very popular data set and


I think among the tutorials they use it as a


As like a template for building your own scripts


um


And that's probably where I got this from but I just by default I go to this data set and use this my template if I'm


building a new data set loading script, so


Come over here. So within the build script. I have a few things here


Jason lines just you can see what was in there. I can actually delete that. I don't need that anymore


So just remove that


What I want to do is create a


Python file and I'm going to name it the same as my data set


So I'm I'm gonna call this the reddit


topics


tar GZ


And that's what I'm gonna call this this data set


Okay


And


This all okay, we're gonna modify a lot of this but for now, I'm not going to touch too much we just want to


Um, once first on the essential things that we need here. So first thing we don't need this


It's like added complexity that isn't necessary


class


We'll call it so reddit


Tar GZ, I suppose it's fine already topics


Builder configs this doesn't matter


This does matter we will mess around that later not now


So what does let's focus on what actually matters right now?


So the here we have this download manager and we're gonna we're gonna look at download manager a bit more in the next video, but


for now download manager is


essentially a hugging face state sets utility that allows us to


given a particular


file either local or on the internet we can download it and


Extract the contents of it. So this is why I formatted the


Date sets file as a tar GZ file


Because I want to use this download and extract function or method


So what we need to do is in your so I'm gonna change this to URL. I'm gonna come up here where we define


URL


Actually move that and here I'm going to replace that with the


Location that I copied earlier. So


See if that actually that won't work. So I need to copy it again. So if I go to here


The repo again. Good zero and build a script the


Location of the compressed file go there and then where it says download


I'm going to copy that link and I'm gonna put in here. Okay, so


with that


Description we can know demo


We'll change the other things later with that


We or this here will kind of almost work so this is just one thing so


We're downloading this URL, but


With squad there were two URLs. Okay, so


If I actually go back a little bit you see that there is these two years one for the training set one for the development


set and


We only have one so we actually


Need to modify this a little bit to deal with just one day. So not two


So here we need to return split generator. We actually just remove this one the validation


split because we just have a training split and the download files


is actually


Not going to be this it's going to be so this is basically going to show us a path to a particular location


Let me show you exactly what it's doing. Okay, so we're going to do from


Transfer no data set. Sorry


Import


Import from data sets utils import download manager


Might not be there. Maybe it's here. Let's see


Okay, it was that so dl manager and i'm just going to


Initialize it. This is kind of happening in the background of our of our builder script


So we don't actually do this in the build script. It just kind of happens


so


We do that and then let's just


Copy what we have elsewhere. So we have the url


And it is this okay, that is the euro


And let's just see what this output so download manager and download


and extract


Your up. Okay. Let's see what we get. Um


We'll call this out


Okay, so we see


We get a file path from that now. Okay


Interesting. So what let's have a look at what is in that file path. So os list out


Okay, so now we can see we actually have that json lines file that we that we put inside our


our compressed tar file


Okay, what what does that mean for us it means we can actually just load that


From here based on what the download manager is giving us. Okay. So this is like a cached location


for our particular data set


So return to the builder script, uh, we have download files


I I don't really like the name so i'm just going to call it path


um


I'm gonna say path here as well remove train


And if we just have a look at the path


Um, it is it is just the directory that contains our data set of json


Okay, or json lines file. So actually what we need to do


Is we need to do like out plus?


Data set dot json out


Okay, this this will give us a full path to our file. So that is what we're going to do here


So, where are we um, so path let's I mean it's a bit easy to read okay come here


Zoom out a little bit


Okay, so it will be path


And then here we have data set dot json out, okay


And yeah, that's how


That's a split generators function here. And what that will do


Is you see that we have this file path here?


That is going to get passed along to this generate examples


function or method and


Is this method that is going to kind of output the data set?


method that is going to kind of output the rows of the data set to us


so


What we need to do is actually


Just use this to kind of read our data set now


It you know just kind of doing that from from stretcher seeing what's happening. It's kind of hard


So let's let's return to that the notebook file and see how we can do that


So we're here we have our


Our let's call this file path now because this is what we've created in the other file. It's a file path


And what we're going to do file path is well first we need to import json because it's a json minus file


so we're gonna have to


read that and


Exactly. We actually want to do this so as fp


I don't think we even need to put encoding there, but we'll put it to be safe


And


What we're going to do is we're going to go through that so for line


In fp it's a json lines file. So there's just lines of data


Each one those lines represents a json object. So we are going to


Oh, we can we can just print it from now in in here


so


Let's put a count on this. So we'll print out a few items, but not too many


so if count is


5 break


Okay, let's see what we get. Okay, cool. So we can see that we we get a few items here. All right


So we're just kind of going through those of red file and we're just looping through and and printing them


So we can do the same over in our other


So we can do the same over in our other file in the builder script so let's come to here


Copy this in now


All of this we can see here. Okay


Some of this we will need not all of it. So let's go ahead and just remove what we don't need


Okay, this is all we need


Um this yield so because this uh generator example is creating a generator function


right


Let's come here remove


Remove parts of this. So the line or the we should call it a record


Is equal to json dot loads


line


Maybe we'll call it object


Okay


And within the object we have a few


A few different key value pairs, right? So what are those?


We can have a look at the the make tar file


File and we have we have all these here so we have subtitle self text


Up vote ratio id and create utc now


We can actually just pass all of these directly onto the next so we can yield all of these


So let me show you what I mean by that


So we come here and you see that we're just yielding and then we're yielding this dictionary type structure for squad


All right, so as we already have that dictionary type structure because we use a json lines file


This is one of the reasons I like using them. So we can actually just do yield key object like that now


Okay, what is um, what is key?


Key is actually the index value or id value if you if you want, but it's an index value


So i'm going to rename it


Index because that makes more sense to me than key


And yeah, here we go. So we have


We have set up here we're going to open the file located or let's let's do that and read the lines. Um,


Load file object or json object


And yeah, we just yield them so what would that do when we are loading


The function or when we are loading the data set over in hugging face data sets


This is going to be the thing that generates all of those all those items


so


What we should do now


is maybe we can


Maybe we can test it and see what happens. Um, it won't work straight away. We'll see


But let's try so what i'm going to do is i'm just going to copy all of this


then i'm going to come over to


Hugging face i'm going to click on my or icon right over here click new data set


I'm going to call it reddit topics


tar gc


create that


And i'm going to come to files i'm going to add a file create new file


And this is just going to be reddit topics


Tar gc. So the exact same file we created before i'm just going to paste all that code in there


Okay, so you see we have all this code. Uh, let's


Let's just remove this


I think it's that important. Uh, it's not squad anymore. So let's just call it reddit topics


Tar gc demo data set


One thing we do need is we need to import json. So that's good. That's already there


Um, we don't need this anymore, but let's keep it in there for now before we start removing everything creating more errors


So let's commit that


And then let's just try and see what happens. Okay, so i'm going to create a new file to test it. So i'm saying


test test data set


And what we're going to do is from data sets import load data set


And the data set we'll just call data load data set


And we can find the data set name over here


So i'm just going to copy click here copy that and there is just one split in this data set. So split equals train


Okay, let's see what happens. Okay, we download the build script so far so good downloads the data


And then we get this. Okay. What is this os error come down here?


cannot find data file


Okay, so we had a look at this so without this


Dot here. Uh, we can see that data file is there so we have our first error, um, which was not on purpose


But that's fine


So the reason we have that is because here I we put a dot i'm not sure why that so let's save that


And actually, let's just edit it in the in the web editor here as well


So let's remove that


commit changes


And then let's try again. Okay, let's come up here. Let's clear everything restart and let's go again


Okay, so now we get this key error. So what does that mean key error context?


Context


Okay, I don't remember putting context anywhere. So let's have a look at the builder script


And if we okay, let's have a look. Okay here we have this so we haven't modified this yet


Now what is this telling us?


Um, it's basically telling the data set builder


Which features to expect in the data set?


So basically down here we're kind of feeding in these these different features. We're feeding in these these records


Each record is a key value pair. So the keys are the feature names and the values are obviously values


Which have a particular data type now?


here


We have the the feature names. So the keys


But they are not aligned to our actual data set. These are using the squad data set key value pairs


So we need to come over to this file and we can


Or get those features specific to our data set from there. So let's take these i'm going to copy them across here


and


all i'm going to do is actually


just


Write those here. Okay, so we have


a subtitle self text


Upvote ratio


Uh, we have oh we have id and we also have another one so let's create another


Well, actually, let's make this one more normal first the id


Is this and then we have one more which is created utc


Okay, now we can try this it's not going to work again, but let's try


Yeah, let's rerun this


See what happens


Okay. So actually it does work, but it's not working in the way that we might expect so


If we have a look at data


and zero


Okay, we have


Subtitle self text and then we come down here. There's a lot in this self text


Um, but so just look at this. So the upvote ratio


Which is a floating point number is now a string


The id that's fine


We should expect that and the created utc which is also a floating point number is now a string as well. So


So there's a bit of an issue here. Basically if we if we go back to our script


When we are feeding the features through this


feature specification


It's seeing that we're saying everything should be a string and it's converting everything into a string


We don't actually want everything to be a string. So what we need to do here is use a specific


apache arrow


Data type identifiers for different things. So for example flow that we have here. Okay, so let's go ahead and have a look at how


What that might be so to find that i'm just going to type like apache arrow data types


Here so apache arrow data types and schemas


schemas maybe


Uh, we come here and we can see we can see a load of these so we have we have integer values unsigned integers


And then we have floats. So


I'm going to say okay single precision floating point type is perfect


Okay, so i'm just going to copy that float 32 and i'm going to put that for created utc and also the operator ratio


Okay, i'm going to save that


I'm going to change a few things that we don't we don't actually need so i'm going to remove this task template because we can't


Do question answering with this data set. Um


Um, no, we can't at least not extractive question answering or can't train with that


for the home page


Let's let's put this I suppose


Okay, supervised keys is none


And what else do we have here? So subscription


Uh, so it's a demo. We know that okay


Okay, let's save this and


And try again. Okay, so i'm going to copy this over into home face come here


Uh not here here


edit


And come here select all paste and I am going to commit those changes


Now let's have a look at what happens if we load the data set so


Come back over here test data set


Uh, let's run this


Let's see what happens


Okay, it loaded well it loaded correctly. That's a good sign come down here


And now we can see that these are no longer strings, but they're actually flowing point numbers. Okay, so


That is


That's everything there are maybe a few aesthetic things to change here. So the


Like the citation


We'll change that up here. I can change this as well, but we're not going to go through that in this


Uh in this video, but I think you want to watch me change citations. So yeah, that's everything


For this video in the next video


What we're going to do is take a look at taking this a a little bit further


And adding more advanced data types like images into our data sets


so


Until then I hope this has been useful


Fast intro to multi-modal ML with OpenAI's CLIP

In this video we're going to have a quick introduction to OpenAI's clip and how we can use it to almost move between the modalities of both language and images.


Now before we dive in let's just quickly understand what clip is.


So it consists of two big models.


In this implementation we're going to be using a vision transformer that will embed images.


And we're going to use a normal text transformer that will embed text.


During pre-training OpenAI trained the model on pairs of images and text and it trained them to both output embedding vectors that are as close as possible to each other.


So the text transformer was trained to output a single embedding 512 dimensional embedding that was as close as possible to the vision transformer's image embedding for the image text pair.


So what that means is that clip is able to take both images and text and embed them both into a similar vector space.


And with that we can do a lot of things.


You can do image and text classification.


You can do image and text search and a huge number of things.


Anything to do with images and text there's a good chance we can do it with clip.


So let's have a look at how we actually use clip.


OpenAI released a GitHub repository OpenAI clip here.


This contains clip but we're not going to use this implementation.


We're actually going to use this implementation of clip.


So this is on Hugging Face.


So we're going to be using Hugging Face transformers and this is still from OpenAI.


It's still clip.


It's just an easy to use implementation of it through the Hugging Face transformers library which is a more standard library for actually doing anything with NLP and also now computer vision and some other things as well.


So to get started I'd recommend you install these libraries.


To install Torch you should probably go through the PyTorch.org instructions rather than following this here.


So go to PyTorch.org and just install PyTorch using the specific install command they use for your platform or your iOS from here.


And then pip install transformers and datasets.


You can still just use this command I'd recommend installing PyTorch from here instead.


Now after that we're going to need our dataset.


So this is just a very simple dataset.


It contains I think just under 10,000 images and we only care about the images here.


So if we have a look we have ImageNet we'll go the first item and we'll just have a look at image.


And we have this Sony radio and we have other things as well.


So if we go ImageNet.


It's 6494.


There's another image here of a dog.


OK just to point out that we have a lot of images in here in the dataset that cover a range of things.


There's not a huge number of different categories here but they have dogs they have radios and a few other things.


Now I'm just going to go ahead and initialize everything.


So there's a few things here.


From transformers we're importing the clip tokenizer.


So the tokenizer is what's going to handle the pre-processing of our text into token ID tensors and other tensors.


We have the clip processor that's like the tokenizer but for images.


So this is actually just going to resize our images into the size that clip expects and also modify the pixel values as well.


And then we have clip model. Clip model is clip itself.


OK so if you have CUDA or MPS if you're on M1 Mac you just set that with this.


OK and then we're ready to actually initialize all of this.


So the model ID is going to be what we saw before.


So you come over here we have the tokenizer clip VIT base patch 32 copy that.


And here we go. OK.


And now we just need to look I'm being told what to do already.


OK so model clip model from pre-trained model ID.


I'm going to I don't normally set device like that.


I don't know if you can. I am going to do it like this.


OK and tokenizer. OK good job. And processor. Cool.


Almost there. It's from pre-trained.


OK and you got a little bit confused. So model ID.


OK that looks good. Let's run that.


OK cool. So now what we're going to do is take a look how we actually create the text embeddings through clip.


So we start with a prompt. I'm going to go with a dog in the snow.


There's not many pictures of dogs in the snow in the dataset but there are some.


And what we need to do is is tokenize the prompt.


Yeah that's true. OK. I'm not going to do it like that.


We're going to go with tokenize prompt and the we need to return tensors using Pytorch.


So we're using we're going to be using Pytorch behind the scenes here.


So make sure we do that. And let's just have a look at what is actually in inputs.


OK so we get the this input ID is tensor so you'll you'll recognize this if you if you use a face transformers before.


These are just the ID token IDs that represent the words from this.


OK. And these this is the attention mask.


Now for us it is going to all be ones but if we had padding in here anything beyond the length of our prompt would become a zero.


Telling the model to not pay attention to to that part of the prompt.


And from there we can process this through clips so we do model get text features I think.


And we pass in those inputs. OK. And let's have a look at the shape of that.


OK so we have a five hundred and twelve dimensional vector.


OK. So that's the text embedding side of things.


Now we need to go ahead and do the image embedding side of things.


OK. So we're going to resize the image first with the processor.


We're not adding any text in here so you can also process text through this processor.


I'm just keeping it separate because it makes more sense to me.


The image should be images actually.


Again we want to return tensors using PyTorch.


OK. And then we can have a look at the I'm going to I'm going to show you the image.


First we have a look at the shape and as well one thing.


So OK I can show you. OK.


OK. In here we actually have this pixel values so we actually need to extract that.


So we're going to put it here. I'm going to move those to the device as well.


I think the device I have set up right now is actually CPU so it doesn't make a difference for me but it's fine.


So let's have a look at the shape.


OK. So you see that we have this 224 by 224 image with three color channels.


So this is just the expected shape that will be consumed by the vision transformer of CLIP.


OK. And to import my PLOTlib.


Pyplot.plt. And I want to show you this image.


So this resize image. So PLT. I'm show image.


And I need to so I need to resize it. Let me show you what I'm actually doing here.


So image.squeeze. Zero. So I'm going to remove that first dimension.


Now I'm going to transpose it. So we put the three color channels at the back.


This is for this is for my PLOTlib to be able to actually show us this.


So I'm going to take that. I'm going to put it here.


OK. And you can see so the minimum maximum color values are all of the color values.


Pixel values are modified when we do this process it through the processor.


So the colors are kind of messed up. But you can see that this is like a resized.


You know what we saw before. OK. So it's a Sony. Just kind of backwards now and flipped.


We can sort it see that it is that Sony radio. So with that we can go ahead and get the image features.


I think it just showed me. Model. Get image features.


So an image. OK. And then let's have a look at the shape.


Cool. OK. So similar to before we have that 512 dimensional embedding vector.


OK. So that's cool. And from here we can we can do a lot of things.


What I'm going to show you how to do is how to kind of search through this or at least compare a small number of images against our prompt


so that we can actually see which one of those images is the most similar to a dog in the snow.


OK. So to do that we're going to want to embed more of these images.


I'm not going to embed loads of them just going to embed 100 images. Nothing nothing crazy.


So we're going to import NumPy as NP. NP random seed.


So this is just so you can replicate what I am doing.


And so this will this will randomly generate a set set set of random numbers. OK.


So the reason I'm doing this is because we want to take a sample out of the data set.


We don't want to have the whole data set. I want it to be at least somewhat random.


So to do that we want to go. So sample.


Indices are going to be equal to NumPy random dot random from zero up to the length of image net.


It's actually plus one. And we need 100 of those.


And then we're going to convert that into a list.


OK. I can just have a quick look at what is in there.


OK. So just all of these all these numbers here. OK.


So yeah. Cool. And if we run it again because we have that random seed set the random set of numbers doesn't change.


And what I'm going to do is just create a list of images using that using those values.


So I for I in sample IDX. OK.


Check. OK. So now 100 images from our data set.


And now we want to just go ahead and literally take everything we've just done and put into a for loop to create the embeddings for all of these images.


OK. So that will look something like this. I'm using TQDM here.


This is just a progress bar so we can see where we are. Batch size saying how many images to perform this for in any one go.


You can increase this if you're on a if you're using a bigger GPU or or whatever else.


Image array I'm setting that to none for now. We initialize that in the first loop.


OK. And then we're just in the same thing as before. So. So from this. So I'm selecting a batch of images based on the on the batch size.


And then where we are processing and resizing the images from that batch we're getting the image features look exactly the same thing.


I think before I actually didn't include pixel values but it's the same thing.


It's just a default argument. Converting into a NumPy array. Did I show you this before? I don't actually think so.


No maybe not. But here the squeeze is very similar. It's the same thing as what I showed you up here.


So we squeeze the first dimension out of that like we did here.


And then we are moving that batch of embeddings to the CPU. If it's not already on the CPU we're detaching it from the gradient like the training graph of PyTorch.


The PyTorch model e.g. clip. And then we're converting into a NumPy array.


OK. And then I'm going to add that batch of embeddings to a larger array of all image embeddings. OK. And that's why the image array comes in.


OK. So let's run that. OK. So we come up here. I made a mistake in the code.


So here I'm actually pulling in the full row or record at any one time. We don't do that. We want the image itself.


OK. So run that again. OK. And now if we check the type of images. Zero. We should see it's a pill image.


Yeah. Cool. Yeah. Pill here. Now we can run this. OK. It won't take long.


And now we have one hundred five hundred twelve dimensional image embeddings from our data set and we can now use them to compare to our initial text embedding and see which one of these matches most closely to that text embedding.


OK. So I'm going to be using dot product similarity. So there's just one thing to be aware of with that. And that is that it considers both the magnitude of the vector and also the angle.


So in this case that will that can throw off our results.


So we should normalize all of the image embeddings so that we are not looking at the magnitude of vectors. And we're only focusing on the angular similarity between our text embedding and these image embeddings.


So to do that we need to. I'll just show you quickly.


So look at the minimum maximum. You know that kind of all over the place. So to normalize we need to do this.


So do image array divided by do numpy.


Linouge dot norm. And here we have the image array.


OK. Axis equals one.


And let me show you what that is. So we have all these numbers and these are basically telling us for each one of these vectors of what should we divide it by in order to bring each of them to within a within a set set.


Within a set magnitude pretty much. So.


Take a look at the shape will be 100. So yeah we do that.


So I think I need to.


Transpose this. OK.


And then so the image array the shape is going to be transposed now so I'm going to transpose it again. Yeah.


Image array equals image array transpose. OK.


Cool. And now if we have a look at the minimum and maximum.


So minimum and maximum we get these values which are more reasonable.


OK. So now what we can do is use dot product similarity to actually compare compare these.


So text embedding I'm going to take the text embedding and similar to before what we did is we need to move it to the CPU.


Detach it from the PyTorch graph and then convert to a numpy array. OK.


Yeah. And then for the scores all we need to do is a numpy dot.


And we are going to put the text embedding followed by the image array. And actually I think I need to transpose this again. So.


Maybe we could have avoided transposing up here.


OK. Yeah. So the scores that we get here we get a single score for every single vector.


As we can see shape 100 and they are the dot product similarity scores.


So what we can now do is sort based on this scores array and just return like the top.


So the top five images and see what the top five most similar images are for our particular query.


OK. So we're going to return the top k. So top k is going to be the five most similar or the five items with the highest score.


And then we want to take the index values using np.org sort.


We're going to add the negative of the scores there and just make sure we take because scores has this here.


So we're actually just taking the let me show you scores 0.


Shape. OK. So it's taking the 100 values there and then I want to take the top k from that.


OK. So what we're left with is these five index values which are essentially indexes of the image embeddings and therefore both the.


And therefore the images that are the most similar to our query.


So we use matplotlib again to to visualize those. So we do for i in ID8.


Let's print the let's print the score first. So scores i.


And actually that would be 0 i and then I'm going to I am going to show and I'm going to PLT show.


OK. Cool. So yeah I mean that's it. So the first first item as we would expect is a dog in the snow.


So after that we get dogs and we get like these snowy areas. The reason for that is that we just don't have any more images of dogs in the snow.


This one I don't know what this is. It's like a toy that maybe it's a dog. Maybe it's a bear.


I'm not sure. But I suppose technically that's like a dog in the snow.


So we have that. So yeah obviously the model is performing pretty well and I think that's very cool that we can do that so easily.


And yeah I mean CLIP is I think an amazing model that we can use to do a load of cool things across both the text and image domain which is super interesting.


And it's definitely like if you think just a couple of years ago this sort of thing was impossible and didn't seem like at least not to this sort of degree of accuracy like it was going to be happening anytime soon.


So this is this is really cool. Here we've obviously shown I showed you how to do like a text to image search.


You can do this like the deep. In reality what we're doing is kind of searching through the vectors.


So it doesn't matter you know which direction you're doing that search. The vectors are all the same.


So if you want to do a text to text search with with CLIP you could. You want to do image to image search you could.


If you want to do image to text or all of those things all at once you could.


It is not as you're searching through vectors. So what is behind those vectors doesn't really matter so much.


OK. So I think that's it for this video. I think CLIP is super interesting and I hope that you do as well in the future or very soon actually.


I'm going to be going into a lot more detail on CLIP.


So if you are interested in that subscribe and click on the little notification button and you will get a notification about that pretty soon.


All You Need to Know on Multilingual Sentence Vectors (1 Model, 50+ Languages)

Today we're going to be having a look at multilingual sentence transformers.


We're going to look at how they work, how they're trained, and why they're so useful.


We're going to be focusing on one specific training method which I think is quite useful


because all it really needs is a reasonably small data set of parallel data which is simply


translation pairs from a source language like English to whichever other language you're using.


So obviously if you are wanting to train a sentence transformer in a language that


doesn't really have that much data, it's particularly sentence similarity data,


this can be really useful for actually taking a high performing, for example, English sentence


transformer and transferring that knowledge or distilling that knowledge into a sentence


transformer for your own language. So I think this will be pretty useful for a lot of you.


And let's jump straight into it.


Before we really get into the whole multilingual sentence transformer part of the video,


I just want to sort of give an impression of what these multilingual sentence transformers


are actually doing. So on here we can see a single English sentence or brief phrase down


at the bottom, I love plants, and the rest of these are all in Italian. So what we have here


are a vector representations of dense vector representations of these phrases. And a monolingual


sentence transformer, which is most of the sentence transformers, will only cope with one language.


So we would hope that phrases that have a similar meaning end up within the same sort of vector


space. So like we have for amo lippiante here, and I love plants, these are kind of in the same space.


A monolingual sentence transformer would do that for similar sentences. So in English, we might


have I love plants and I like plants, which is actually what we have up here. So this here is


Italian for I like plants. And we would hope that they're in a similar area, whereas irrelevant or


almost contradictory sentences we would hope would be far off somewhere else like our vector over here.


So that's how obviously a monolingual sentence transformer works. And it's exactly the same for


a multilingual sentence transformer. The only difference is that rather than having a single


language, it will comprehend multiple languages. And that's what you can see in this visual. So


in this example, I have I love plants and amo lippiante, they have the same meaning,


they have the same meaning just in different languages. So that means that they should be as


close together as possible in this vector space. So here we're just visualizing three dimensions.


In reality, it'll be a lot more. I think most transforming models go with 768 dimensions.


But obviously we can't visualize that. So we have 3D here. So we want different languages or similar


sentences from different languages to end up in the same area. And we also want to be able to


represent relationships between different sentences that are similar. And we can kind of see that


relationship here. So we have mi piacere e le piante and amo lippiante and I love plants are all


kind of in the same sort of area. Mi piacere e le piante, so I like plants, is obviously separated


somewhat, but it's still within the same area. And then in the bottom left down there we have


un cane arancione, which means I have a orange dog. So obviously, you know, that's really nothing


to do with I love plants. Although I suppose you could say it's you're talking about yourself, so


maybe it's a little bit similar, but otherwise they're completely different topics. So that's


kind of what we want to build. Something that takes sentences from different languages and maps them


into a vector space, which has some sort of numerical structure to represent the semantic


meaning of those sentences. And it should be language agnostic. So obviously we can't, well,


maybe we can train on every language. I don't know any models that are trained in every single


language, but we want it to be able to comprehend different languages and not be biased towards


different phrases in different languages, but to have a very balanced comprehension of all of them.


Okay. So that's how the vectors should look. And then, okay. So how, what would a training data


for this look like? And what are the training approaches? So like I said before, there's two


training approaches that I'm going to just briefly touch upon, but we're going to focus on the


latter of those. So the first one that I want to mention is what the M-U-S-E, MUSE or Multilingual


Universal Sentence Encoder Model was trained on, which is a multitask


translation bridging approach to training. So what I mean by that is it uses two or uses a dual encoder


structure and those encoders deal with two different tasks. So on one end you have the


parallel data training. So when we say parallel data, these are sentence pairs in different


languages. So like we had before, we had the Amalepiante and Isle of Plants, which is just the


Italian and English phrases for Isle of Plants. So we would have our source language


and also the translation or the target language. It's probably a better way to put translation now.


So we have the source and translation, that's our parallel data set. And what we're doing is


optimizing to get those two vectors or the two sentence vectors produced by either one of those


sentences as close as possible. And then there is also the source data. So we basically have


like sentence similarity or NLI data, but we have it just for the source language. So we have source


sentence A and source sentence B. And we train on both of these. Now this is, it works and that's


good, but obviously we train on a multi-task architecture here and training on a single task


in machine learning is already hard enough. Training on two and getting them to balance


and train well is harder. And the amount of data, at least for Muse and I believe for,


if you're training using this approach, you're going to need to use a similar amount of data,


is pretty significant. I think Muse is something like a billion pairs, so it is pretty high.


And another thing is that we also need something called hard negatives in the training data


in order for this model to perform well. So what I mean by hard negative is, say we have our,


you know, we have our source sentence A here and we have this source B, which is like a similar


sentence, a high similarity sentence. They mean basically the same thing. We'd also have to add a


source C and this source C will have to be similar in the words that uses to source A,


but actually means something different. So it's harder for the model to differentiate between


them. Again, the model would have to figure out, you know, these two sentences are not similar,


even though they seem similar at first, but they're not. So it makes the task, the training


task harder for the model, which of course makes the model better. So that is training approach


number one. And we've mentioned the parallel data there. That's the data set we're going to be using


for the second training approach. And that second training approach is called multi-lingual


knowledge distillation. So that is a mouthful and takes me a while to write down, sorry. So


multi-lingual knowledge distillation. So this was introduced in 2020 by, you know, who we mentioned


before, the sentence transformers people, Nils Reimers and Irenia Gurevich. And the sort of


advantage of using this approach is that we only need the parallel data set. So we only need those


translation pairs and the amount of training data you need is a lot smaller. And using this approach,


the sentence transformers people have actually trained sentence transformers that can use all


more than 50 languages at once. And the performance is good. It's not just that they, you know,


managed to get a few phrases correct. The performance is actually quite good. So I think,


you know, it's pretty impressive. And, you know, the training time for these is super quick,


as we'll see. And like I said, it's using just translation data, parallel data, which is reasonably


easy to get for almost every language. So I think that's pretty useful. Now, well, let's have a look


at what that multi-lingual knowledge distillation training process actually looks like. So it's what


we have here. So same example as before. I've got, I like plants this time and mi piaccia


no le piante, which is again, the same thing in Italian. Now we have both of those. We have a


teacher model and a student model. Now, when we say knowledge distillation, that means where you


basically take one model and you distill the knowledge from that one model into another model


here. The model that already knows some of the stuff that we want, that we want to distill


knowledge from is called the teacher model. Okay. Now the teacher model in this case is going to be


a monolingual model. So it's probably going to be a sentence transformer. That's very good


at English tests only. And what we do is we take the student model, which is going to be,


it doesn't have to be a sentence transformer. It's just a pre-trained transform model. We'll be using


XLM Roberta later on, and it needs to be capable of understanding multiple languages. Okay. So


in this case, we feed the English sentence into both our teacher model and student model. And


then we optimize the student model to reduce the difference between the two vectors output


by those two models. And that makes the student model almost mimic the monolingual aspect of the


teacher model. But then we take it a little further and we process the Italian or the target language


through the student model. And then we do the same thing. So we try to reduce the difference


between the Italian vector and the teacher's English vector. And what we're doing there is


making the student model mimic the teacher for a different language. Okay. So through that process,


you can add more and more languages to a student model, which mimics your teacher model, which,


yeah, I mean, it seems at least really simple just to think of it like that in my opinion anyway,


but it works really well. So it's a very cool technique in my opinion. I do like it. So just


a more visual way of going through that. We have these different circles, they represent different


language tasks or different languages, but pretty similar or the same task in each one of those.


We have our monolingual teacher model and that can perform on one of these languages, but fails


on the others. We take that monolingual model or our teacher model, and then we also take a pre-trained


multilingual model. So the important thing here is that it can handle new languages. Like I said,


with Excel learning Roberta, this is our student. We perform multilingual knowledge distillation,


meaning the student learns how the teacher performs well on the single task by mimicking


its sentence vector outputs. The student then performs this mimicry across multiple languages.


And then hopefully the student model can now perform across all of the languages that we are


wanting to train on. That's how the multilingual knowledge distillation works. Let's have a look


at that in code. Okay. So we're in our code here. And the first thing I'm going to do is actually


get our data. So in the paper that introduced the multilingual knowledge distillation,


Reimers and Guravich use the focus partly on this TED subtitles data. So we know TED talks,


they're just low talks where people present on a particular topic, usually pretty interesting.


And those TED talks have subtitles in loads of different languages. So they scraped that subtitle


data and use that as sentence pairs for the different languages. Okay. So that's the parallel


data. Now, what I'm going to do is use Hug and Face Transformers to download that. So we just


import datasets here. Sorry, I said Hug and Face Transformers actually Hug and Face datasets here.


So import datasets and I'm going to load that dataset and just have a look at what the structure


of that dataset is. So it's the TEDMortar and I'm just getting the training data here. You see in


here we have this features, translations and talk name. Now it's not really very clear.


But inside the translations data, we have the language tag. So these are language codes,


ISO language codes. If you type that into Google, they'll pop up. If you don't know which one,


which are which. And below we also have in here, it's not very clear again. So if I come here,


we have translations and each one of those corresponds to the language code up here.


Okay. So if we came here, we see EN English and we find it here. Okay. And then we also have


talk name. It's not really important for us. So we can get the index of our English text. We need


to extract that for our source language. So we extract that, we get a number four. So we're


going into those language pairs, finding EN. And then we use that index to get the corresponding


translation, which is here. And then we'd use that to create all of our pairs. Now here,


I've just created loads of pairs. This first one, so this is English to Arabic. But if we have a


load, there's actually loads of pairs here. So we have 27 in total, which is obviously quite a lot.


We're probably not going to use all of those. I mean, you could do if you wanted, it depends on


what you're trying to build. But I think most of us are probably not going to be trying to build


some model that crosses all these different languages. So what I'm going to do is just


initialize a list of languages that we would like to train on. So we're going to be feeding all of


this into a sentence transformer class called parallel sentences dataset. And that requires


that we one, separate our pairs using a tab character. And two, keep all those pairs separated


in different GZIP files. So that's why I'm using this particular structure. So data pre-processing


steps here, I'm just running through them quickly because I want to focus more on the actual


sentence transformer training part. So run that, we can, well, it's actually going to take a moment.


So let me skip forwards. Okay. And then we want to see how many pairs, well, I just want to see,


we don't have to do this. I want to see how many pairs we have for each language.


And you see here we have about 200,000 for each of them. The German one is slightly less.


Okay. And then let's have a look at what those source and translations look like. So here we


have applause and applause. Now I think that's Italian, it seems so. But here we can see, okay,


the end of the talk ends in applause. Obviously the subtitles say applause, or hopefully it ends


in applause. And then we just have the tab character and that separates the source language,


English in this case, from the translated language. Now, okay. Now what we want to do is save that


data. So we sort all that in these dictionaries. Okay. So initialize dictionary here and access


them here. So we have enit, es, ar, fr, and de. And now I'm just going to save them. So run this.


That will save. And what I'll do is, sorry, OSLister. So we can see what is in there.


Where is it? It's data. Just data. Is that right? Okay. And then we have these five files. Okay.


Now let's continue. So now what we want to do is, okay, we have, that's our training data. It's


ready or mostly ready before we feed it into the Sentence Transformer's parallel sentences


dataset object later on. So, okay, let's leave that for now and move on to the next step, which is


choosing our teacher and student models. So, you know, I already mentioned before, we want our


student model to be capable of multilingual comprehension. So what I mean by that, or


not just what I mean, but one big component of that is, can the Transformer tokenizer deal with


different languages? In some cases they really can't. So let me show you what the Debert tokenizer


does with these four different sentences. So we'll just loop through each one. So four text in


sentences. And what I'm going to do is just print. I'm going to print the output of Debert


tokenizer. And if I tokenize that text, now what does it give me? Okay. So what we have here, okay,


English, of course, is fine. But the tokenizer or the vocabulary of the tokenizer is, I think,


roughly 30,000 tokens. Okay. And most of those are English based. Okay. Some, like you can see here,


that it has picked up some Chinese characters because it does, you know, other languages do


feed into it a little bit because it's just, you know, all the data is pulled from the internet,


other bits do get in there, but it's mostly English. So that's why we see, okay, we have these


unknown characters. Now, as soon as we have an unknown character in our sentence, the tokenizer,


or no, sorry, the transformer is read against the wood to understand, you know, what is in that


position? What is that unknown token supposed to represent? In the case of, you know, I think of it


as it's like, you know, when you're a kid in school and they had those, you know, had like a


paragraph and you had to fill in the blanks, right? So you had a paragraph and occasionally


in a couple of sentences, there'll be a couple of blank lines where you need to, you know,


guess what the correct word should be. If you only have a couple of those blanks, you know,


as a person, you can probably guess accurately. And the same for Bert. Bert can probably guess


accurately what the occasional unknown token is. But you know, if you're a kid in school,


you can guess what the actual unknown token is. But if in school they gave you a sheet and they


said, okay, fill out these blanks. And it was actually just a paragraph of blank and you had


to guess it correctly, you've probably, I don't know, I think your chances are pretty slim of


getting that correct. So the same is true for Bert. Bert, for example, in our Georgian example


though. So the tokenizer from Bert is not suitable for non-Latin character languages whatsoever.


And then it does know some Greek characters here and maybe it knows all of them. So I suppose Greek


feeds into Latin languages a bit more than Georgian or Chinese, but it doesn't know what


to do with them. They're all single character tokens. And the issue with single character


tokens is that you can't really encode that much information into a single character.


Because that, you know, if you have 24 characters in your alphabet, that means you have 24 encodings


to represent your entire language, which is not going to happen. So, you know, that's also not


good. So basically don't use a Bert tokenizer. It's not a good idea. What you can do is, okay,


how's this XLM token or tokenizer? Now, XLM is trained for multilingual comprehension.


It uses a sentence piece transformer, which uses byte level logic to split up the sentences or


the words. So it can deal with tokens it's never seen before, which is pretty nice. And the


vocabulary size for this is not 30k. I think it's 250k. It could be off a few k there, but it's


around that mark. And it's been trained on many languages. So it's obviously a much better option


for our student model. So let's have a look at how we initialize that. So this XMR model is just


coming from transformers. Okay. So I need to convert that model from just a transform model


into an, or initialize it as a sentence transformer model using the sentence transformers library.


Okay. So from sentence transformers, I'm going to import models and also sentence transformer.


So XMR, so this is going to be our actual transformer model. We're going to write models.transformer.


Sentence transformers under hood uses hugging face transformers as well. So we would access this


as the normal model identified that we would with normal hugging face transformers,


which is XMR Roberta base. Okay. As well as that, we need a pooling layer.


So we write models.pooling. And in here, we need to pass the output embeddings dimension. So it's


this get word embedding dimension for our model. And also what type of pooling we'd like to do.


We have max pooling, CLS token pooling, and what we want is a mean pooling. So is pooling


mode mean tokens equals true. Okay. So that two components of our sentence transformer.


And then from there, we can initialize our students. So student equals sentence transformer.


And we're initializing that using the modules, which is just a list of our two components. So


XMR followed by pooling. And that's it. So let's have a look at what we have there.


Okay. We can just ignore this top bit here. We just want to focus on this. So you see,


we have our transforming model followed by the pooling here. And we also see that we're using


the mean tokens pooling set to true, rest of them are false. Okay. So that's our student model


initialized. And now what we want to do is initialize our teach model. Now, the teach model


let me show you, you just have to be a little bit careful with this. So sentence transformer.


So maybe you'd like to use one of the top forming ones, which a lot of them are the old models.


So these are monolingual models, all MPNet base V2.


And okay, let's initialize this and let's see what is inside it. Okay. So we have the transformer,


the pooling as we had before, but then we also have this normalization layer. So the outputs from


this model are normalized. And obviously if you're trying to make another model mimic the


normalization layer outputs, well, it's not ideal because the model is going to be trying to


normalize its own vectors. So you don't really want to do that. You want to choose a model.


You either want to remove the normalization layer or just choose a model that doesn't have


normalization layer, which I think is probably the better option. So that's what I'm going to do.


So for the teacher, I'm going to use a sentence transformer. I'm going to use paraphrase models


because these don't use normalization layers. Distill Roberta base V2. Okay, let's have a look.


Okay. So now you see we have the transformer followed directly by the pooling. Now another


thing that you probably should just be aware of here is that we have this max sequence length here


is 512, which doesn't align with our paraphrase model here. But that's fine because I'm going to


limit the maximum sequence length anyway to 512. So that's fine. But I'm going to limit the


the maximum sequence length anyway to 250. So, you know, don't, you know, it's not really an issue,


but just, you know, look out for that if you're training your own models. This one's on 384.


So none of those align. But yeah, just be aware of that, that the sequence lengths might not


align there. So we've, okay, so we have our, we've sort of formatted our training data.


We have our two models, the teacher and the student. So now what we can do is prepare that


data for loading into our training process or fine tuning process. So as I said before,


we're going to be using the parallel sentences, sorry, from sentence transformers import


parallel sentences data set. And first thing we need to do here is actually initialize the object.


And that requires that we pass the two models that we're training with, because this kind of handles


the interaction between those two models as well. So obviously we have our student model,


which is our student. And we have the teacher model, which is our teacher. Alongside this,


we want batch size. I'm going to use 32, but I think actually you can probably use higher batches


here. Or you probably should use higher batches. 64 is one that I see used a lot in these training


codes. And you also use embedding cache called true. Okay. So that initializes the


parallel sentences data set object. And now what we want to do is add our data to it. So


we need our training files. So training files equal to OS list that we did before.


I think it's in the data file, in the data directory.


Yeah. So that's all we want. And what I'll do is just for F in those training files,


I'm going to load each one of those into the data set object. Print F and data dot load data.


I need to make sure I include the path there, followed by the actual file name.


I need to pass your max sentences, which is the maximum number of sentences that you're


going to take from that load data batch. So basically the maximum number of sentences we're


going to use from each language there. Now I'm just going to set this to 250,000,


which is higher than any of the batches we have. That's fine. I don't think, I mean,


if you want to try and balance it out, that's fine. You can do that here.


And then the other option is where we set the maximum length of the sentences that we're going


to be processing. So that is max sentence length. And I said before, look, the maximum we have here


is 256 or 512. So let's just trim all of those down to 256. Okay. That will load our data.


And now we just need to initialize a data loader. So we're just using PyTorch here. So run from


torch utils.data, input data loader. Loader is equal to data loader. Pass out data.


We want to shuffle that data. And we also want to set the batch size, which is same as before, 32.


Okay. So model is already, data is ready. Now we initialize our loss function. So from


sentence transformers again, dot loss, losses. Yep. Import MSE loss.


And then loss is equal to MSE loss. And then here we have model equals student model. Okay. So we're


only optimizing our student model, not the teacher model. The teacher model is there to teach our


student, not the other way around. Okay. So that's everything we need ready for training. So let's


move on to the actual training function. So we can train, I'm going to train for one epoch,


but you can do more. I think in the actual zone, in the other codes that I've seen that do this,


they were training for like five epochs. But you even just training on one epoch,


how you actually get a pretty good model. So I think you don't need to train on too many, but


obviously, you know, if you want better performance, I would go with the five that I've seen in the


other codes. So we need to pass our train objectors here. So we have the data loader and then loss


function. Now I want to say, okay, how many epochs? Like I said before, I'm going to get with one


number of warm up steps. So before you jump straight up to the learning rate that you,


you will select in a moment, do we want to warm up first? Yes, we do. I'm going to warm up for


10% of the training data, which is just length of the loader and multiplied by 0.1.


Okay. And from there, where do you want to save the model? I'm going to try,


I'm going to save it in xml-ted.


Now optimizer parameters.


So we have a, we're going to set a learning rate of 2e to the minus 5,


epsilon of 1e to the minus 6. And we're also going to set correct bias equal to false.


Okay. There the optimizer parameters. And then we can also save the best model. Save the best


model equal to true. And then we run it. Okay. So run that. It's going to take a long time. So


I'm going to actually going to stop it because I've already run it. And let's have a look at the,


actually evaluating that and have a look at the results. Okay. So I just have this notebook where


I've evaluated the model. So I'm using this STS sentence textual similarity benchmark dataset,


which is multilingual. I'm getting the English data and also the Italian. And you can see


they are similar. So the zero, so each row in the English dataset corresponds to the other


language datasets as well. So in here, sentence one in the English means the same thing as sentence


zero in the Italian. Okay. Same sentence two, also same similarity score. So first thing we do is


normalize that similarity score. And then we go down a little bit. So we reformat data using


sentence transformers input example class. And through this I've created three different evaluation


sets. So we have the English to English, Italian to Italian, and then English to Italian.


And then what we do here is we initialize a similarity evaluator for each of these datasets.


Again, we're using sentence transformers, just makes life a lot easier. We initialize those and


then we can just pass our model to each one of those evaluators to get its performance. So here


81.6 on the English set, 74.3 and 71 here. Now I just trained on one epoch. If you want better


performance, you can train on one epoch and you should be able to get more towards 80% or maybe


a little bit higher. So pretty straightforward and incredibly easy. And then here I'm just


I wanted to compare that to the student before we trained it. So I initialize a new student and had


a look and you can see the evaluation is pretty low. So for English, 47.5. Italian, actually 50%,


surprisingly. Although it's already a multilingual model. So it does make sense that you can understand


Italian. And then from English to Italian, it rarely drops down to 23. So that's it for this


video. I think it's been pretty useful. At least for me, I can kind of see where you can build a


sentence transformer in a lot of different languages using this, which is, I think, really cool.


And will probably be useful for a lot of people. So I hope you enjoyed the video.


Build a Custom Transformer Tokenizer - Transformers From Scratch #2

Hi, welcome to the video. We're going to have a look at how we can build our own tokenizer


in transformers from scratch. So this is the second video in our transformers from scratch


series. And what we're going to be covering is that the actual tokenizer itself.


So we've already got our data so we can cross off now onto the tokenizer. So let's move over


to our code. So in the previous video, we created all these files here. So these are just a lot


of text files that contain the Italian subset from the Oscar dataset.


Now let's maybe open one, ignore that, and we just we get all this Italian. Now each sample


in this text file is separated by a newline character.


So let's go ahead and begin using that data to build our tokenizer. So we first want to get a


list of all the paths to our files. So we are going to be using pathlib. You could also use


oslist there as well. It's up to you. Import path. So from pathlib, import path.


I'm using this one because I don't know, I've noticed that people are using this a lot at


the moment for machine learning stuff. I'm not sure why you would do it over oslist there,


but it's what people are using. So let's give it a go, see how it is. So we have this


and we just want to create a string from each path object that we get. So for x in,


and then in here, we need to write path and in here we just want to


basically tell this where to look. So we're using path here and we're just in the same directory.


So it's not, we don't really need to do anything here. That's fine. And then at the end,


we are going to use glob here. I think this is why people are using this.


And we just create like a wildcard, like we want all text files in this directory. So we just write


that. Now let's do that. I look at the first five and see that we have our text files now.


So that's good. And what we can now do is move on to actually training the tokenizer. So the


tokenizer that we're going to be using is a byte level, byte pair encoding tokenizer or BP tokenizer.


And essentially what that means is that it's going to break down our text into bytes. So


with most tokenizers that we probably use, unless you've used this one before, then


we've used it before. We tend to have like unknown tokens. So like for BERT, we use sentence piece


encodings and we have to have this unknown token for when we don't have a token for a specific


word, like for some new word. Now with the BPE tokenizer, we are breaking things down into bytes.


So essentially we don't actually need an unknown token anymore. So that's I think pretty cool.


Now to use that, we need to do from tokenizers. So this is another Hugging Face package. So


maybe you might need to install that. So pip install tokenizers.


And you want to do byte level BP tokenizer like that. Okay. Now we take that and we're going to


initialize our tokenizer. So we just write that. That's our tokenizer initialized. We haven't


trained it yet. To train it, we need to write tokenizer train. And then in here, we need to


include the files that we're training on. So this is why we have that pass variable up here. So this


is just a list of all the text files that we created, which are all separated by newline


characters. Each sample is separated by a newline character. Now the vocab size,


we're going to be using a Roberta model here. And I think the Roberta model, typical Roberta model


vocab size is 50k. Now, you can use that if you want this up to use, but I'm going to stick with


the typical BERT size just because I don't think we need that much. We're just figuring things


out here. So this is going to mean less training time. And that's a good thing, in my opinion.


We'll set the min frequency. So this is saying, what is the minimum number of times you want to


see a word or a part of a word or a byte? So it's kind of weird with this tokenizer


before you add it into our vocabulary. So that's all that is. Okay. And then we also need to


include our special tokens. So we're using the Roberta special tokens here. So we write


special tokens. And then in here, we have our start sequence token, which I'm going to


put this on the new line. So not like that, like this. So we have this start sequence token,


the padding token, end of sequence, which is like this, the unknown token, which with it being a


byte level encoding, you'd hope it doesn't need to use this. And then we have our start sequence


token. So it doesn't need to use this very much, but it's there anyway. And the mastern token. So


that's everything we need to train our model.


And one thing I do remember is if you train on all of that, all of those files, it takes a really


very, very long time, which is fine if you're training it overnight or something, but


that's not what we're doing here. So I'm just going to shorten that to the first 100 tokens.


And maybe I'll train it after this with the full set. Let's see. So I will leave that to train for


a while and I'll be back when it's done. Okay. So it's finished training our tokenizer and we can


go ahead and actually save it. So I'm going to import OS, just soon so I can make a new directory


to store the tokenizer files in. And a typical Italian name, or so I've been told, is Filiberto,


which fits really well. But so this is our Italian BERT model name, Filiberto. So that is our


new directory. And if we just come over to here, we have this working directory, which is what I'm


in. And then we have this new directory, Filiberto, in here. That's where we're going to save our


tokenizer. So we just write tokenizer, save model. And here we can see here, we can do save or save


model. Save just saves a JSON file with our tokenizer data inside it. But I don't think


that's the standard way of doing it. I think this is the way that you want to be doing it.


And we're saving it as Filiberto, like that. So we'll do that. And we see that we get these


two new files, vocab.json and merges.txt. Now, if we look over here, we see both of those.


And these are essentially like the two steps of tokenization for our tokenizer.


So when we feed text into our tokenizer, it first goes to merges.txt. And in here, we have


characters, words, so on. And they are translated into these tokens. So these are characters on the


right, tokens on the left. So we scroll down, we can see different ones. We can keep going.


So here we have Zeone. That's like, although my Italian is very bad, that is like the English


T-I-O-N. So T-I-O-N. And we would say something like attention, right? Italians have the same,


but they have like attention. So that's what we have there. So it's part of a word and it's


pretty common. And that gets translated into this token here. Now, after that, our tokenizer


moves into vocab.json. And I don't know why it's sorted at the bottom there. Go to the top.


If I clean this up quickly, we can see we have a JSON object. It's like a dictionary in Python.


And we have all of our tokens and the token IDs that they will get translated into. So if we


scroll down here, we should be able to find, was it VA, I think. Okay, so VA1.


Which is our Zeone into this token here. And then that eventually gets converted into this token ID.


So that's our full tokenizer process.


Just open that file back up. If we wanted to load that, we would do that like we normally would


with transformers. So we'd start from transformers. And then we would go to transformers.


So we'd start from transformers, import Roberta. So we're using a Roberta tokenizer here.


So we're both at tokenizer. We can use either the Roberta tokenizer or the fast version. It's up to you.


And we just initialize our tokenizer.


Like that. We front retrained. And in here, rather than putting a model name from the Huginface website,


we would put the path, the local path to our directory, our model directory.


So it's filiberta for us. And then we can use that to begin encoding text. So


go ciao, come va, which is like hi, how are you?


If we write that, we can see that we get these are the tokens here. I wonder if we did a 10.


So I'll do that. I'll try in a minute. So we have the starter sequence token here.


And the sequence token here. So the S and the S like that. So we have those at the start and end of each


sequence. And we can also add padding in there. So padding equals max length. And also max length


needs to have a value as well. So max length, 12. And then we get these padding tokens, which are the ones.


So that's pretty cool. And I just want to, purely out of curiosity, anything else?


So we have potenzione. Let's see if we recognize the number there. So no, we don't.


So I suppose this is probably the full word. In fact, it is. So this is the full token here.


If we just do this, maybe we will get, I can't remember what number it was.


The 322. Maybe that's right. I'm not sure. But anyway, that's how everything works.


So that's it for this video. In the next video, we will take a look how we can use this tokenizer


to build out our input pipeline for training our actual Transformer model.


Streamlit for ML #2 - ML Models and APIs

Okay, in the last video, we had a look at how to build what you can see on the screen right now.


A very simple interface using Streamlit.


Now, what we want to do in this video is go through how we actually build the smart part


behind the Open Domain Q&A system that we're going to put together here.


So, I said before, there are a few components to Open Domain Q&A.


We're going to stick to the first two for now.


So the Vector Database, which we're going to use Pinecone for,


and the Retriever Model, which we are going to download from Hug & Face Model Hub.


And we're going to use the Sentence Transformers library to actually implement that.


Now, the first thing we are going to want to do is create our Vector Database.


So our index.


Now, to do that, there are three parts or three steps we need to take.


First, we need to download our data.


We're going to be using the Squad Dataset from Hug & Face Datasets.


Then we want to encode those vectors, encode those paragraphs,


or what we call context, into context vectors.


And we use Sentence Transformers and a Retriever Model for that.


And then the next part is uploading or pushing all of those vectors into our Pinecone Vector Database.


So, to do all of that, we're just going to very quickly go through that code,


because it is a lot and I don't want to focus on it too much.


So here we have a script.


I'm going to maybe zoom out a little bit so you can see.


So the first thing we do is import everything.


You don't need TQDM here, but you can pip install TQDM if you do want to use that.


So we are from Datasets.


So this is Hug & Face Datasets.


You will need to install this.


So that is just a pip install Datasets.


We're going to first initialize our retriever model.


We're using the Pinecone MPNet Retriever Squad 2 Retriever Model.


So this is a retriever model that is based on the MPNet model from Microsoft.


And it's been trained on the Squad 2 Dataset.


And first thing we need to do is initialize our connection to Pinecone.


So this is where we're going to store all of our vectors.


Now to do that, you do need an API key.


I wouldn't write it in your code, but I'm going to just do that for the sake of simplicity here.


So I'm going to go to this app.pinecone.io.


And this is free by the way.


You don't have to pay anything.


So we just go to app.pinecone.io.


And then you will have to sign up.


So you create an account.


I already have one, so I don't need to worry about that.


And I have this default API key over here.


I could use that.


And yeah, I'm just going to use that so we can see the key if we want.


I want to zoom in a little bit.


Of course, it's a little bit bigger.


So we can see that.


And we can see the value there.


We can just copy it or we just press over here and copy that across.


And then I'm just going to copy that across.


And then I'm just going to paste it in here.


Okay.


Now this script, by the way, I will leave a link to this in the description.


So you can just download it instead of writing it all out.


Because this isn't essential to our app.


It's just how we build our...


We encode all of our contexts and actually saw them in our vector database.


So we have that.


We have the cloud environment that we're using there.


Switch this back to the app.


We want to check if the index already exists.


So I'm going to create this QA index.


Now, actually, you can see in mind I already have it because I've run this code already.


So QA index already exists.


So it's not going to create a new index.


And instead, it's just going to connect that index here.


Right.


So we just connected or we created our index, our vector database index.


And now what I want to do is I'm going to switch back to our data.


And I'm going to run through that.


So I'm going to load the data set and the squad data set from Hugging Face.


Now I'm going to use a validation split because the model has been trained on the training data.


But squad, I want to make it a little bit hard.


So we're going to use a validation split that hasn't seen before.


I'm removing any unique or duplicate context in there.


So zoom out a little bit here.


Squad depth, we're using this filter.


So this is all Hugging Face data sets syntax here.


And then we're encoding it.


So this model.encode, so this is our sentence transformer.


We're encoding it to create a load of sentence vectors for our context.


And we're converting these to lists because we are going to be pushing these


through an API request to Pinecone.


Again, we need a list, not a NumPy array or otherwise you're going to get an error.


OK.


Then back to the Pinecone side of things.


We want to create a list of, it's basically a list of tuples.


And those tuples include the ID of each context.


So there's a unique ID for each context.


We want the vector or the encoding, the context vector.


And then we also have this dictionary here.


Now this is metadata.


So metadata in Pinecone is like any other information about your vectors that you want to include.


And this is really good if you want to use metadata filtering, which is super powerful in Pinecone.


And I definitely want to leave the option open later on.


I'm not sure if we'll use it or not.


We'll probably put something in there just so we can play around with it.


Now that creates the format that we need to upset everything, which means just like push or upload


everything to Pinecone.


So then I do that in chunks of 50 at a time.


Just makes things a little bit easier on the API aggressor rather than sending everything at once.


OK.


So that's like how we create the index.


So now what we're going to do is actually integrate that a little bit in our app.


So let's switch back to our app here.


Let's move this over here.


Let's view it.


So first, let's just remove this.


We don't need that.


OK, save will automatically reload.


So first thing we want to do here is let's initialize the Pinecone connection.


So I'm going to just take.


Let's just take this part of the code.


Just copy it and then we'll remove what we don't need in a minute.


And so we don't need we do need sentence transformers in a minute.


We don't need data sets.


We do need Pinecone.


So actually here we're initializing our retriever model.


It's the same as what we did before.


So we do want to keep that in there.


Make that bigger.


API key again, just store this somewhere else.


Or if you are using Streamlit Cloud, they have like a secrets management system.


And it's something we'll look at in the future for sure.


But for now, I'm just putting it in here.


So we have our API key environment and we're just doing the same thing we did before.


But actually, we don't want to create an index.


We're assuming we've already created an index if we're in our app.


So we're just going to connect to it.


OK, so with that, we've got our API key.


We've kind of set up the like the back end part of our app.


Like the smart part that's going to handle the open the main Q&A.


But it's going to be a little bit slow.


And we will have a look at how to solve that pretty soon.


But for now, what we're going to do is actually just implement this.


And we're going to actually query and see what we return.


So I'm going to save this.


We won't see anything change in our app now other than the fact that it takes longer to load


because it's downloading the driver model.


That's the main part of the slowness here and then obviously connecting to Pinecone.


Also takes a second as well.


So for now, we're going to deal with how slow it is.


But we will fix that pretty soon.


And now I actually want to do is I would say, OK, if the query is not empty,


because by default it is empty, that's why we add that in there.


So I'm going to actually remove this.


Enter.


If it is not empty, so if query is not equal to nothing,


we're going to query Pinecone for whatever is in that query.


So the first thing we need to do is create our context vector.


So I'm going to write XQ, just shorthand for context vector.


It's pretty standard, especially if you use FICE before they tend to use this.


And I say I said context vector, I meant query vector.


So we're going to do model and code.


And we need to put this in square brackets and we have query.


OK, and I'm going to convert that to a list.


OK, so this is going to create our query vector.


Let's write it down.


Create query vector.


And then the next thing we want to do is query Pinecone with this vector.


So to do that, we want to write first list.


Get relevant context.


And we're going to store these in XC.


So like query like context vector, similar thing to the


query vector that we used before with XQ.


But this time I'm going to write index.query.


And we're going to pass XQ, so our query vector.


And we're going to say how many results we want to return.


Now, later on, we're going to use Streamlit, a little like a slider bar to decide how many


we would like to return.


But for now, we will hard code it.


And another thing that we want to include here is we want to tell Pinecone to return


the metadata because by default, it will not return metadata.


So return metadata equals true.


So these are like the extra little bits I mentioned before.


So included the title.


So like the topic, Wikipedia topic that the context is coming from and also the text itself.


OK, so we're going to return the relevant context and then we're going to look through


each of those.


Now, when we do this, there's a particular format that we need to follow.


So our context are actually going to be stored for context in XC results.


And results is going to return a list and we just want the first item in that list.


The reason it returns a list is because if you are querying Pinecone with multiple queries,


it will return lists of your answers for each query.


But in this case, we are only ever going to query with one query vector.


So we always enter at position zero here and then in there we will have all of our returned


matches inside this matches key value value.


So for context in there, all we're going to do is write st.write context.


And then we want to go into the metadata that we were returning.


And we have title and text here.


We don't want title, we want text.


OK, so let's save that and check that actually works.


So again, this is going to take a little while to load because we're initializing like the


full pipeline of our vector database and the tree model.


Every time we run this, it's downloading the full tree model, which takes quite a bit of time.


OK, so this is just rerun our app.


Now we can say who are the Normans?


OK.


Again, it's going to reload everything, so it's going to take a while.


We're going to fix this in the next video.


So we should be returning five contexts and if we scroll down, we can see we have these


five paragraphs.


Each one of these paragraphs is a single context.


So we could maybe we can respect the element.


OK, so we can see down here.


Pretty horrific to look at.


So if I zoom up, we can see each one of these is a single one of our contexts.


Right.


These here.


Cool.


So I think that is that's it for this video.


So now we have these backend working and the next one, what we'll do is we'll do a little


bit of a fix and the next one, what we'll do is fix this issue with it taking forever


to load, reload everything every time, which is actually super easy.


But we'll make a big difference to our app.


So thank you very much for watching and I will see you in the next one.


Unicode Normalization for NLP in Python

Okay, so we're going to take a look at Unicode normalization.


Unicode Normalization for NLP in Python

Unicode normalization is something that we use when we have those weird font variants


Unicode Normalization for NLP in Python

that people always use on the internet.


Unicode Normalization for NLP in Python

So if you've ever seen people using those odd characters, I think they use it to express


Unicode Normalization for NLP in Python

some form of individuality or to catch your attention.


Unicode Normalization for NLP in Python

And then we also have another issue where we have weird glyphs in text.


Unicode Normalization for NLP in Python

And this is more reasonable because it's actually a part of language.


Unicode Normalization for NLP in Python

That's like little glyphs, so you have the accents above the E's and stuff in Italian


Unicode Normalization for NLP in Python

or Spanish.


Unicode Normalization for NLP in Python

And those little glyphs all together, they're called diacritics.


Unicode Normalization for NLP in Python

And whenever we come across diacritics or that weird text, we can get issues when we're


Unicode Normalization for NLP in Python

building models.


Unicode Normalization for NLP in Python

The issues with the weird text is obviously if we have someone who's got hello world in


Unicode Normalization for NLP in Python

normal text and we're comparing it to someone who's got hello world in some weird text with