Training and Testing an Italian BERT - Transformers From Scratch #4

Hi, welcome to the video.


So this is the fourth video in a Transformers


from Scratch mini series.


So if you haven't been following along,


we've essentially covered what you can see on the screen.


So we got some data.


We built a tokenizer with it.


And then we've set up our input pipeline


ready to begin actually training our model, which


is what we're going to cover in this video.


So let's move over to the code.


And we see here that we have essentially everything


we've done so far.


So we've built our input data, our input pipeline.


And we're now at a point where we have a data loader,


PyTorch data loader, ready.


And we can begin training a model with it.


So there are a few things to be aware of.


So I mean, first, let's just have a quick look


at the structure of our data.


So when we're training a model for mass language modeling,


we need a few tensors.


We need three tensors.


And this is for training Roberta, by the way, as well.


Same thing with Bert as well.


We have our input IDs, attention mask, and our labels.


Our input IDs have roughly 15% of their values masked.


So we can see that here we have these two tensors.


These are the labels.


And we have the real tokens in here, the token IDs.


And then in our input IDs tensor,


we have these being replaced with mask tokens,


the number fours.


So that's the structure of our input data.


We've created a Torch data set from it


and use that to create a Torch data loader.


And with that, we can actually begin


setting up our model for training.


So there are a few things to that.


We can't just begin training straight away.


So the first thing that we need to do


is create a Roberta config object.


And the config object is something


that we use when we're initializing a transformer


from scratch in order to initialize it


with a certain set of parameters.


So we'll do that first.


So we want from transformers import Roberta config.


And to create that config object, we do this.


So we do Roberta config.


And then in here, we need to specify different parameters.


Now, one of the main ones is the voc up size.


Now, this needs to match to whichever voc up size


we have already created in our tokenizer


when we're initializing it.


In our tokenizer, when building our tokenizer.


So I mean, for me, if I go all the way up here to here,


this is where I created the tokenizer.


I can see, OK, it's this number here.


So 30,522.


So I'm going to set that.


But if you don't have that, you can just write tokenizer voc


up size.


So here.


And that will return your voc up size.


So I mean, let's replace that.


We'll do this.


Now, as well as that, we want to also set this.


So max position embedding.


And this needs to be set to your max length plus two


in this case.


So max length is set up here.


So where is it?


Max length here, 512.


Plus two because we have these added special tokens.


If we don't do that, we'll end up with a index error


because we're going beyond the embedding limits.


Now we want our hidden size.


So this is the size of the vectors


that our embedding layers within Roberta will create.


So each token, so we have 514 or 12 tokens.


And each one of those will be signed a vector of size 768.


This is the typical number.


So that's the originally came from the BERT based model.


Then we set up the architecture of the internals of the model.


So we want the number of attention heads,


which I'm going to set to 12.


And also the number of hidden layers, which I...


So the default for this is for Roberta, 12.


But I'm going to go with six for the sake of keeping train


times a little shorter.


Now we also need to add type, vocab, size, which is just one.


So that's the different token types that we have.


We just have one.


Don't need to worry about that.


OK, so that's our configuration object ready.


And we can import and initialize a Roberta model with that.


So we went from transformers.


This is kind of similar to what we usually do.


Import Roberta.


And we're doing this for massed LL.


So MLM, right?


So we're training using MLM.


So we want Roberta for massed LL.


And we initialize our model using that Roberta for massed LL


object.


And we just pass in our config.


And this will...


That's right there, initialize our Roberta model.


So that's a plain Roberta model, randomly initialized weights,


and so on.


And now we can move on to setting up everything


for training.


So we have our model.


Now we need to prepare a few things before we train it.


First thing is we need to decide which device we're


going to be training on.


So whether that's CPU or a CUDA-enabled GPU.


And to figure out if we have that, we write...


Well, we can write torch CUDA is available.


So write this.


And for me, it is.


So the typical way that you would decide whether you're


using CUDA or CPU or the typical line of code that will decide it


for you is you write device and you do torch.CUDA,


or torch device, sorry.


And then you write CUDA inside here.


If it's available, otherwise we are going to use torch device CPU.


Now, CPU takes...


Yeah, it's just takes a really long time.


So if you are using CPU, you have to leave it overnight,


for sure.


Maybe even longer.


Even if it's just like a little bit of data, it takes so long.


So but hopefully you have a GPU.


If not, just you're going to have to be patient.


That's all.


Or if you could maybe try and use Google Colab,


but you have to use a premium version because otherwise it's


just going to shut off after like an hour or two.


I don't know.


I don't really use it.


So I don't know how long it will train for before just deciding


that it's done.


And the GPU is also not that good anyway.


So, yeah.


However, you can do it.


And then after that, we want to move our model to our device.


So whether it's GPU or CPU, we move over there.


We're going to get a really big output now.


So it's just our model.


So this is like the structure of our model.


So we can see a few interesting things.


We've got Roberta for MLM.


We have the Roberta model.


And then inside that we have our embeddings.


And then we have our 12.


Did I say 12?


I think it was six.


Six encoders should be.


Yeah.


So it goes from 0 to 5.


It's our six.


And then we have the outputs here.


And then our final bit, which is the language modeling


head, the MLM head.


So that's cool.


Now we need our optimizer.


So from transformers, import AdamW,


which is Adam with weight and decay.


And what we're going to do is just


going to activate the training mode of our model.


It's going to give us loads of output again.


So just, yeah.


You know what?


Maybe I can just, let's just remove that.


There we go.


Easier.


And then our optimizer is going to be AdamW.


We need to pass in our model parameters.


And we need a learning rate.


So from, I mean, I don't usually use Roberta.


But looking online, this looks like a reasonable learning


rate.


I think you can go from sort of here to, I think,


from what I remember down to like here.


That's the sort of typical range.


But obviously, it's going to depend on how much data


you have.


And we don't want to do that.


How much data you have and loads of different things, right?


So that's what I'm going to go with.


And that should be pretty much it.


So that's our setup.


Now we're just going to create our training loop.


Now for the training loop, we want to import TQDM.


So we can see how far through we are.


We're going to train for two epochs.


And we're going to initialize our loop object using TQDM.


So TQDM.


We have our data loader.


What is the name of that data loader?


I'm not sure.


Data loader.


Cool.


And we set leave equals true.


But I need that.


Sorry.


I need that in the same style.


So for batch in loop.


And then here, we run through each of the steps


that we're going to perform for every single training loop.


So the first thing we do is initialize the gradient


in our optimizer.


So zero grad.


So the reason we do this is after the first loop,


our optimizer is going to be assigned


a set of gradients, which we're going


to use to optimize our model.


And on the next loop, we don't want those residual gradients


to still be there in our optimizer.


We want to essentially reset it for the next loop.


So that's what we're doing here.


Then we want our tensors.


So we have input IDs.


And that is going to be batch input IDs.


And we also want to move that over to our GPU or CPU


if you're on that.


And this is pretty much the same for our three.


So mask labels.


And this is just attention mask.


So we've extracted our tensors.


And we just need to feed them into our model now.


So we're going to get outputs from the model.


We just do model.


Input IDs, attention mask, which is going to be equal to mask.


And our labels equal to labels.


So everything has been fed into our model.


We have our outputs.


Now we need to extract a few things from the output.


So we need the loss.


So we write loss equals outputs.loss.


And from that, we want to calculate


all of the different parameters in our model.


We need to calculate the loss for each one of those parameters.


So we do this loss.backwards to backpropagate


through all of those different values and get that loss.


After we've done that, we use our optimizer, take a step,


and optimize all those parameters based on that loss.


Then that's everything we need to train the model.


And there's just a few things.


So for the progress bar, I just want


a little bit of information there just so I know what's going on.


And I just write loop set description.


And I just want to print out the epoch.


So write that.


And then I want to set the postfix as well.


So loop.set postfix.


And here I just want to see the loss.


So we'll just do loss item like that.


So that should be everything.


Yeah, let's run that, see what happens.


Hopefully it should work.


No, it didn't work.


OK.


Let me see.


Oh, no, it's a CUDA error.


So probably just need to refresh everything.


I hate CUDA errors.


One moment.


OK, so finally figured it out.


Took so long.


So a few tips anyway.


When you do get a CUDA error, switch your device to CPU,


and then re-run everything.


And you should get a more understandable error.


So if we come down here, I've changed it to CPU.


We see that we get an index error.


Scroll down, index out of range itself.


So the reason for this is, so you get this error


if you don't add the extra two tokens onto the end of here.


But we added them.


So I was pretty confused about that.


And then it took me a really long time


to realize that this argument is wrong.


And there should be an S on the end.


So that was the error.


So yeah, super, super cool.


That was literally it.


It took me so long to figure that out.


But now we have it.


That's good.


We just need to run everything again.


So I'm just going to run through everything.


Remove this cell here where I changed it to CPU,


because I don't need it now.


And just re-execute all that.


OK, so we're back.


And we've finished training our model now.


Now, it has taken a long time.


It's a few days later.


And I made a few changes during training as well.


So this definitely wasn't the cleanest training process,


because I was kind of updating parameters as it was going along.


So initially, well, first, we've trained


for like three and a bit epochs.


And I've trained on the full data set as well.


So if I come up here, I think, do I print out


how much data it was?


Maybe in another file.


So if we come down here, so yeah, there's a lot more data here.


So we have 200, no, 20.


Let me think, 2 million.


OK, so 2 million samples in that final run.


And initially, when we started training,


we started with a learning rate of 1e to the minus 5.


Now, I looked into this a little bit.


And it just was not really moving.


And I'll show you in a minute.


So for the second epoch, I moved it down to 1e to the minus 4.


Or moved it up, sorry, to 1e to the minus 4.


And that moved, started moving things a lot quicker.


So that was good.


And then in total, like I said, it was three and a bit epochs.


Other than that, I didn't really change anything.


The only thing I did was I trained for one epoch


at a time because I wanted to see how the results were


looking after each epoch.


And that was quite interesting.


So let me show you that.


OK, so this is after the first epoch.


So here, what I'm doing is I've got this fill,


which is a pipeline fill object.


And I'm entering chow and then putting in now mass and then


va.


And I wanted to say chow, come here, va.


And in the middle, I wouldn't have to predict come.


Now, this is after the first epoch.


And we can see it's not, yeah, it's just putting


like random characters.


So question mark here, three dots here, chow and chow again


here.


Kind of weird.


So yeah, not the best, right?


Then we move on to the second epoch.


And it's getting, well, it's still rubbish.


At least it's got words.


So like here, we have a word, chow, kiva, or chiva.


Kek, oh, chow, kiva.


I don't know if that's the way I always, the C-H in Italian,


I always get messed up.


If there's any Italians watching, I'm sorry.


Chow, kiva.


At least we're getting words.


But none of these, so it doesn't make any sense.


OK, so no, still not good.


Now, if we come across again, so this is this one.


Yeah, this one.


Now we get it.


So the rest of them are nonsense.


OK, so the four here, ignore them.


However, at the top, we get this score of 0.33.


And we get chow, kiva.


So that's what we wanted.


So that's good.


It means it's working.


This was after the third and a bit epoch.


Let me show you the loss function as well.


So this, I know this is really messy.


So here we have our, I don't know why this one's so short,


actually.


Why is that one so short?


Strange, maybe I didn't.


Yeah, for the last one, it doesn't


look like I finished training for the full epoch.


So I thought I did.


Maybe something happened.


I'm not sure.


But fine, that's what it is.


That's fine.


So the first set of training I did was here.


And you see in the middle, my computer


went to sleep for a bit overnight


because it was just so loud.


So I turned it off for a bit.


And then continue going down.


Now, this first epoch is when we were at 1.0 or 1e to the minus 5.


And then here, I was testing the 1e to the minus 4.


And you can see straight away, it goes down way quicker.


So I was like, OK, we're going to go with that.


It's clearly a lot better.


And then continued over here, the next epoch,


and then the final one here, which didn't


seem to change much anyway.


But there was still a pretty clear difference.


So that's the loss over time.


And yeah, I mean, we've seen the results from that.


So now we have that.


Let's move on to actually testing the model.


So I'm going to bring Lara in.


I'm going to just open the file.


OK, so this is the testing we're going to do.


So we're using the file mask.


We've got this pipeline.


Sorry, fill mask.


I've got this pipeline.


And what I'm going to do is just get Lara


to come in and some Italian sentences


and just add this random mask token in


and see if the results are bearable or not.


So let's see.


So I will see you in a minute.


This is Lara.


She can speak Italian.


So she's going to go through this and test it a few times


and hopefully say it's good.


Let's see.


Hopefully.


Ciao.


OK, so all you need to do is we have a sentence here.


And you just write some Italian.


And then for one of the words in there,


we want to replace it with this text here.


And then that's going to mask that word.


And then the model is going to try and predict what is there.


And hopefully it will predict.


So just write some Italian phrase.


Not too difficult yet.


And see.


So I don't have to write, ciao, va?


No, no, no, no.


You write.


Just write a sentence.


And we'll do that.


Buongiorno, gente.


No, buongiorno.


Maybe a few words there.


Can I put comma or dots?


Yeah, you can put comma.


Buongiorno, come va?


OK.


And then, so which word should we cover?


Come?


And then.


OK, so just cover it with the mask.


And see what it says.


So not this.


I seem to read on these as well.


OK, so let's give it a moment.


Chi va?


Yeah, but the second one, come va?


Come va.


Almost there.


Does chi va mean anything?


Like who?


Yeah, it's like, is there someone?


But I understand because I'm Italian,


but I don't think that we don't usually say that.


I don't think.


I'm going to take that as fine.


I'm going to take that as it's good.


So let's do it again, maybe.


Yeah.


Try another one.


Oh, wait, actually, what about these ones?


Cos'va, no, non'va definitely not right.


No.


Che'va?


No, OK.


But it's just like after buongiorno,


I wouldn't expect.


Chi va?


Chi va.


Yeah, that's OK.


So you can just put another one like where we put Phil again.


You can write another sentence.


So we're here.


So we can write.


Yeah, and then what do you want to replace?


Incontriamo, maybe?


Or dove as well.


OK.


Yeah.


So which one?


You decide.


It's fine.


Incontriamo is interesting.


OK, let's try.


Yeah, yeah.


Oh, look at that.


Yeah, that's good.


Dove ci vediamo oggi pomeriggio?


Dove ci incontriamo oggi pomeriggio?


Dove ci siamo oggi pomeriggio?


No.


Dove ci troviamo oggi pomeriggio?


Dove ci ritroviamo?


Yeah, that's quite good.


OK.


Should we try with dove?


Like using the same phrase.


Try.


You can do it.


You can control Z, right?


Yeah.


OK, let's run it.


Dove, the second one.


Come ci incontriamo quando ci incontriamo is good.


Si.


That's cool.


Yeah, cosa, se.


Let's try another one.


Cosa, pripare, che cena stacera.


Let's remove pripare.


Control Z.


Go.


Run it.


Cosa fare per cena stacera, that's good.


Cosa serve per cena stacera, yeah.


Cosa aspetta per cena stacera, ni.


No.


Cosa succede per cena stacera, ni.


Cosa vedere per cena stacera, ni.


So I didn't find what we said before,


which was cosa, pripare per cena stacera.


Yeah.


But does it make sense?


Yeah, it makes sense.


Cosa fare per cena stacera, cosa serve per cena stacera, si.


Try something hard, like grammatically difficult.


Mm.


Cosa succederebbe.


Is it hard?


No, I'm thinking, I don't, when it's like that, you know,


I don't know, like, doesn't come to my mind.


Cosa se avesse scelto un altro giorno?


Okay.


And then what should we replace?


Avessimo.


Avess...


How do you say?


Si.


What does that mean?


If we had.


Avessi, avessi, avessero.


Not, not the same one.


No, but it's good because avessimo, it's for third person plural.


It's like we had.


Avessi is third person singular.


So if he had or she had.


Yeah.


Chosen something different.


What does this actually mean, cosa?


So what would have happened if we had chosen another day?


Ah.


So the first one, se avesse scelto, it will be the third person.


Uh-huh.


Se avesse scelto, it will be the first person.


So if I had chosen another day.


Se avessero, it's second person plural.


So it will be if they had chosen another day.


Se ave...


This one no.


This is good, se venisse scelto, as well.


No, maybe no.


But the first three are very good.


I have an idea.


So now if we change to se...


So if we put se loro, so if we specify the person, maybe we'll take the correct one.


So if we put se loro.


And then we expect it to say avessemo?


Avesero.


So let's run it.


You see?


That's cool.


That's very good.


And then the other one, se loro anno, is right?


Ave anno?


I mean, I'm saying...


Well, the verb is incorrect, but...


Yeah, it's in the wrong place, but it's saying the right...


Like the meaning is correct, but the grammar, it's not correct.


Okay.


Right, okay.


Yeah.


It's cool.


You're welcome.


Not happy it actually worked, because I wasn't sure if I could just...


Wait, what? Chao came over, but that was all I tested it with, so I was a little bit worried it might not do anything else.


But thank you.


Bye.


Okay, so I think that's a pretty good result.


So, I mean, that's pretty much everything we needed for building our model, our transform model.


Although I do want to... So we're going to do one more video after this, where we're going to upload our model to the Hugging Face model hub.


And then what we'll be able to do is actually download it directly from Hugging Face, which I think will be super cool to do that and figure out how we actually put all that together.


So, yeah, I think good result.


I'm pretty happy with that.


And thank you for watching.


Choosing Indexes for Similarity Search (Faiss in Python)

Hi, welcome to the video.


I'm going to take you through a few different indexes in FIAS today.


So FIAS for similarity search.


And we're going to learn how we can decide which index to use based on our data.


Now, these indexes are reasonably complex,


but we're going to just have a high level look at each one of them.


At some point in the future, we'll go into more depth for sure.


But for now, this is what we're going to do.


So we're going to cover the indexes that you see on the screen at the moment.


So we have the flat indexes, which are just the plain and simple,


nothing special going on there.


And then we're going to have a look at LSH or locality sensitive hashing,


HNSW, which is hierarchical navigable small worlds.


And then finally, we're going to have a look at an IVF index as well.


So first thing I'm going to show you is how to get some data for following through this.


So we're going to be using the SIFT 1M dataset,


which is one million vectors that we can use for testing similarity.


Now, there's a little bit of code, so I'm just going to show it to you.


So we have here, we're just downloading the code.


There'll be a notebook for this in the description as well.


So you can just use that and copy things across.


But we're downloading it from here and this will give us a tar file.


So we download that.


And then here, all we're doing is extracting all the files from inside that tar file.


And then here, I'm reading everything into the notebook.


So inside that tar file, we'll get these FVEX files


and we have to open them in a certain way, which is what we're doing here.


So we're setting up the function to read them, sorry, here.


And then here, I'm reading in two files.


So we get a few different files here.


So I'm sorry, this should be SIFT.


So we get the base data, which is going to be the data that we're going to search through.


And then we also have query data here.


And then what I'm doing here is just selecting a single query,


a single vector to query with rather than all of them,


because we get quite a few in there.


And then here, we can just see.


So this is our query vector that gets Q.


And then we also have WB here, which is going to be the data that we'll index and search through.


And we can see some of it there as well.


So that's how we get data.


Let's move on to some flat indexes.


So what you can see at the moment is a sort of a visual representation of a flat L2 index.


Now up here, this is what we're doing.


So we're calculating, we have all these points.


So these are all of the WB points that we saw before.


And this is our query vector.


And we just calculate the distance between all of those.


And then what we do is just take the top three.


So the top K in reality, but in this case, it's top three.


Now, we also have IP.


So we have both L2 distance and IP distance as well.


IP works in a different way.


So we're using a different format to actually calculate the distance or similarity there.


So it's not exactly as you see it here.


But before we write any code, just want to say with flat indexes, they are 100% quality.


And typically what we want to do with FI's and similarity search indexes is balance the


search quality versus the search speed.


Higher search quality, usually slower search speed.


And flat indexes are just pure search quality because they are an exhaustive search.


They check the distance between your query vector and every other vector in the index,


which is fine if you don't have a particularly big data set or you don't care about time.


But if you do, then you probably don't want to use that because it can take an incredibly


long time.


If you have a billion vectors in your data set and you do 100 queries a minute, then


as far as I know, it's impossible to run that.


And if you were going to run that, you'd need some pretty insane hardware.


So we can't use flat indexes and exhaustive search in most cases.


But I will show you how to do it.


So first, I'm just going to define dimensionality of our data, which is 128, which we can see


up here, 128.


I'm also going to say how many, so how many results do we want to return?


I'm going to say 10.


Okay.


We also need to import FI's before we do anything.


And then we can initialize our index.


So I said we have two.


So we have FI's index flat 02 or IP.


I'm going to use IP because it's very slightly faster.


It seems from me testing it, it's very slightly faster, but there's hardly any difference


in reality.


So initializes our index and then we want to add our data to it.


So we add WB and then we perform a search.


So let me create a new cell and let me just run this quickly.


And what I'm going to do is just time it so you can see how long this takes as well.


So I'm going to do time and we're going to do index or sorry, DI equals index search.


And in here we have our query vector and how many samples we'd like to return.


So I'm going to go with K.


So that was reasonably quick and that's because we don't have a huge data set and we're just


searching for one query.


So it's not really too much of a problem there.


But what I do want to show you is, so if we print out I that returns all of the IDs or


the indexes of the 10 most similar vectors.


Now I'm going to use that as a baseline for each of our other indexes.


So this is, like I said, 100% quality and we can use this accuracy to test out other


indexes as well.


So what I'm going to do is take that and convert it into a list.


And if we just have a look at what we get, we see that we get a list like that.


And we're just going to use that, like I said, to see how our other indexes are performing.


So we'll move on to the other indexes.


And like I said before, we want to try and go from this, which is the flat indexes, which


is 100% search quality to something that's more 50-50.


But it depends on our use case as well.


Sometimes we might want more speed, sometimes higher quality.


So we will see a few of those through these indexes.


So we start with LSH.


So a very high level.


LSH works by grouping vectors in two different buckets.


Now what we can see on the screen now is a typical hashing function for like a Python


dictionary.


And what these hashing functions do is they try to minimize collisions.


So collision is where we would have the case of two items, maybe say these two, being hashed


into the same bucket.


And with a dictionary, you don't want that because you want every bucket to be an independent


value.


Otherwise, it increases the complexity of extracting your values from a single bucket


if they've collided.


Now LSH is slightly different because we actually do want to group things.


So we can see it as a dictionary.


But rather than where before we were avoiding those collisions, you can see here we're putting


them into completely different buckets every time.


Rather than doing that, we're trying to maximize collisions.


So you can see here that we've pushed all three of these keys into this single bucket


here.


And we've also pushed all of these keys into this single bucket.


So we get groupings of our values.


Now when it comes to performing our search, we process our query through the same hashing


function and that will push it to one of our buckets.


Now in the case of maybe appearing in this bucket here, we use Hamming Distance to find


the nearest bucket and then we can search or we restrict our scope to these values.


So we just restricted our scope there, which means that we do not need to search through


everything.


We are avoiding searching through those values down there.


Now let's have a look at how we implement that.


So it's pretty straightforward.


All we do is index, we do vise index LSH.


We have our dimensionality and then we also have this other variable which is called nbits.


So I will put that in a variable up here.


Do nbits and what I'm going to do is I'm going to make it d multiplied by 4.


So nbits we will have to scale with the dimensionality of our data which comes into another problem


which I'll mention later on which is the curse of dimensionality.


But I'll talk more about it in a moment.


So here we have nbits and then we add our data like we did before and then we can search


our data just like we did before.


So time and we do d pi equals index search and we are searching using our query, our


search query and we want to return 10 items.


So quicker speed, see here.


And what we can also do is compare the results to our 100% quality index or flat index and


we do that using numpy in 1D baseline i.


Okay so I'm just going to look at it visually here so we can see we have quite a lot of


matches here.


So plenty of trues, couple of falses, true, false, false, false.


So these are the top 10 that have been returned using our LSH algorithm and we're checking


if they exist in the baseline results that we got from our flat index earlier and we're


returning that most of them are present in that baseline.


So most of them do match so it's a reasonably good recall there.


So that's good and it was faster.


So we've got 17.6 milliseconds here.


How much did we get up here?


We got 157 milliseconds.


So slightly less accurate but what is that 10 times faster so it's pretty good.


And we can mess around with n bits.


We can increase it to increase the accuracy of our index or we decrease it to increase


the speed.


So again it's just trying to find that balance between them both.


Okay so this is a graph just showing you the recalls with different n bit values.


So as we sort of saw before we increase the n bits value for good recall but at the same


time we have that curse of dimensionality.


So if we are multiplying our dimensionality value d by 8 in order to get a good recall


then if we have a dimensionality of 4 that's not a very high number so it's going to be


reasonably fast but if we increase that to dimensionality for example 512 that becomes


very, very complex very quickly.


So you have to be careful with your dimensionality.


Lower dimensionality is very good for LSH otherwise it's not so good.


You can see that here so at the bottom here I've used this is on the same data set so


an n bits value of d multiplied by 2 with LSH it's super fast.


It's faster than our flat index which is what you would hope but if we increase the n bits


value quite a bit so maybe you want very high performance then it gets out of hand very


quickly and our search time just grows massively.


So you kind of have to find that balance but what we got before was pretty good.


We had a d multiplied by 4 I think and we got reasonable performance and it was fast


so it's good.


And that also applies to the index size as well so lower n bits size index size isn't


too bad.


With higher n bits it's pretty huge so also something to think about.


Now let's move on to HNSW.


Now HNSW is what first part of it is NSW which is Navigo Small World Graphs.


Now what makes a graph small world it essentially means that this graph can be very large but


the number of hops so the number of steps you need to take between any two vertexes


which is the points is very low so in this example here we have this vertex over here


and to get over to this one on the opposite side we need to take 1, 2, 3, 4 hops and this


is obviously a very small network so it doesn't really count but you can see this sort of


behaviour in very large networks so I think in 2016 there was a study from Facebook and


at that point I don't remember the exact number of people that they had on the platform but


I think it's in the billions and they found that the average number of hops that you need


to take between any two people on the platform is like 3.6 so that's a very good example


of a Navigo Small World Graph.


Now hierarchical NSW graphs which is what we are using they're built in the same way


like a NSW graph but then they're split across multiple layers which is what you can see


here and when we are performing our search the path it takes will hop between different


layers in order to find our nearest neighbour.


Now it's pretty complicated and this is really I think over simplifying it a lot but that's


the general gist of it I'm not going to go any further into it we will I think in a future


video and article.


Now let's put that together in code.


So we have a few different variables here we have M which I'm going to set to 16 and


M is the number of connections that each vertex has so of course that means greater connectivity


we're probably going to find our nearest neighbours more accurately.


EF search which is how what is the depth of our search every time we perform a search


so we can set this to a higher value if we want to search more the network or a low value


if we want to search less of the network obviously low values can be quicker high value can be


more accurate.


And then we have EF construction now this similar to EF search is how much of the network


will we search but not during the actual search during the construction of the network.


So this is essentially how efficiently and accurately are we going to build the network


in the first place.


So this will increase the add time but the search time it makes no difference on so it's


good to use a high number I think for this one.


So initialise our index and we have is FICE index HNSW flat so we can use different vectors


here that we can I think PQ PQ there and essentially what that's going to do is make this search


faster but slightly less accurate.


Now this is already really fast with flats and that's all we're going to stick with but


again like I said we will return to this at some point in future and cover it in a lot


more detail for sure.


So dimensionality we need to pass in our M value here as well.


Now we want to apply those two parameters so we have EF search which is obviously EF


search and then we also have HNSW the obviously the EF construction.


So that should be everything ready to go and all we want to do now is add our data.


So index.add WB.


Okay now like I said we have that EF construction we use a reasonably high value so you can


see this is already taking a lot longer than the previous indexes to actually add our vectors


into it but it's still not going to take that long.


And then once it is done we are going to do our search just like we did every other time


so we have DI equals search sorry index.search and we are going to pass in our query and


also K.


Okay so 43.6 seconds to add the vectors there so a fair bit longer but then look at this


super fast like that 3.7 milliseconds.


So much faster than the last one I think the last one was 16 milliseconds right.


Okay this is a flat index 157 LSH we have 17.6 okay so really quick which is cool.


But how's the performance so let's have a look.


Okay so we get quite a few false's here and only a couple of true's so okay it's not so


great it was really fast but it's not very accurate but fortunately we can fix that so


let's increase our EF search I'm going to increase it a fair bit let's go 32.32.


This is probably I would imagine more than enough to get good performance so run this


and run this.


Okay and now we see we get pretty good results.


Now the war time is higher so it's just a case of balancing it because this is now higher


than LSH but what we can do is increase EF construction time the value of EF construction


increases or decreases depending on what you want.


There's a lot of flexibility with this and it can be really fast this is HNSW is essentially


one of the best performing indexes that you can use if you look at the current state of


the art a lot of them are HNSW or they're based on HNSW in some way or another.


So these are good ones good ones to go with you just need to play around them a little


bit.


So this is a few of the performance I found using the same data set but I'm messing around


so we have the EF construction values down here so start with 16 over here up to 64 EF


search values over here and our M values over here.


And we've got pretty good recall over 64 on the EF construction so EF construction is


a really good one to just increase because it doesn't increase your search time which


is pretty cool I think.


And then here is the search time again for HNSW M and EF search obviously I didn't include


EF construction there because it doesn't make a difference.


And that's this is the one thing with HNSW the index size is absolutely huge so that's


just one thing to bear in mind the index size can take a lot of memory but otherwise really


really cool index.


And then that leaves us on to our final index which is the IVF index and this is super popular


and with good reasons it is very good.


So the inverted file index is based on essentially clustering data points so you see here we


have all of these different data points the little crosses and then we have these three


other points which are going to be our cluster centroids.


So around each or based in each of our centroids we expand a catchment radius around each of


those and as you can see here where each of those circles collides it creates the edge


of what are going to be our almost like catchment cells.


This is called a Voronoi diagram or try it's a really hard word Dirichlet tessellation


I don't know if that's correct but it sounds I think it sounds pretty cool so I thought


I'd throw that in there.


So we create these cells in each one of those cells any data point within those cells will


be allocated to that given centroid and then when you search within a specific cell you


pass your XQ value in there and that will be compared the XQ value will be compared


to every single cluster centroid but not the other values within that cluster or the other


clusters only the cluster centroids and then from that you find out which centroid is the


closest to your query vector and then what we do is we restrict our search scope to only


the data points within that cluster or that cell and then we calculate the nearest vector


so at this point we have all the vectors only within that cell and we compare all of those


to our query vector.


Now there is one problem with this which is called the edge problem.


Now we're just showing this in two-dimensional space obviously in reality for example the


data set we're using we have 128 dimensions so dimensionally the edge problem is kind


of complicated when you think about it in the hundreds of dimensions but what this is


is so with say with our query we find our query vectors right on the edge of one of


the cells and if we sell n probe value so I mentioned n probe here that's how many


cells we search if that is set to one it means that we're going to restrict our search to


only that cell even though if you if you look at this we have two or we have I'm trying


to think so this one for sure is closer to our query vector than any of the magenta data


points and possibly also this one and this one but and maybe even this one but we're


not going to consider any of those because we're restricting our search only to this


cell so we're only going to look at you know these data points and also these over here


so that's that's the edge problem but we can get around that by not just searching one


cell but by searching quite a few so in this case our n probe value is eight and that means


we're going to search eight of the nearest centroids or centroid cells and that's how


IVF works let's go ahead and implement that in code so first thing we need to do is sell


n list value which is the number of centroids that we will have within our within our data


and then this time so this is a little bit different we need to set the the final vector


search that we're going to do so we're this is kind of split into two different operations


right so we're searching based on clusters and then we're actually comparing the full


vectors within the selected clusters so we need to define how we're going to do that


final that final search between our full vectors and our query vector so what we do is write


vice so do index flat we're going to index five p you can use l2 as well we set our dimension


it so we're just initializing a flat index there and then what we're going to do is feed


that into our IVF index so our IVF index is vice index IVF and flat because we're using


the flat indexes the flat vectors there we need to pass our quantizer so the this step


here the other step to the search process the dimensionality and also our n list values


of how many cells or clusters we're going to have in there and with this because we're


clustering data we need to do something else so in fact let me show you so if we write


index dot is trained we get this false if we wrote off any of our other indexes this


would have been true because they don't need to be trained because we're not doing clustering


or any other form of training or optimization there so what we need to do is we need to


train our index before we use it so we write index train and we just pass all of our vectors


into that but it's very quick so it's not really an issue and then we do index add pass


our data and then what we do one thing so I want to show you we have our n probe value


we'll search with one for now so we search one cell and to search we write di as we have


every other time search execute okay okay so I mean super fast 3.32 milliseconds I think


that's maybe the fastest other than how bad performing or low quality hsw index so let's


see how how that's performed so you write mp dot in on d baseline hi you see it's not


too bad to be fair like 50 50 almost so that's actually pretty good but what we can do if


we want it to be even better is we increase the n probe value so let's go up to four so


that's increased the wartime quite a bit so from like three to 125 which is now super


slow actually but now we're getting perfect results we can maybe decrease that to two


so now it's faster that could have been a one-off sometimes occasionally you get a really


slow search and just happens sometimes so this is so we set n probe to super fast and


super accurate so that that's a very good index as well so these are the stats I got


in terms of recall and search time in milliseconds for different n probe values and different


endless values so again it's all it's just about balancing it again index size the only


thing that affects your index size here is obviously the size of your data and the endless


value but you can increase the endless value loads and the index size hardly increases


so this is like increasing by 100 kilobytes per like double of the endless value so this


is very it's like nothing so that's it for this video and we covered quite a lot so I'm


gonna leave it there but I think these all these indexes are super useful and quite interesting


and figuring out just playing around with them like you see I've done loads with these


these graphs just seeing what is faster what is slower what where the good quality is I'm


just playing around the parameters and seeing what you can get out of it is super useful


for actually understanding these now what I do want to do going forward is actually


explore each one of these indexes in more depth because you've only covered them like


very very very high level at the moment so in future videos articles we're going to go


into more depth and explore them a lot more so that we pretty interesting I think so that's


it for this video thank you very much for watching and I will see you in the next one


Training BERT #4 - Train With Next Sentence Prediction (NSP)

Hi and welcome to the video.


Here we're going to have a look at


how we can use NSP


or Net Sentence Prediction


to train a BERT model.


Now in


a previous video I covered


how NSP works but I didn't


really cover how you actually train


a model using it. So


that's what we're going to do here.


So we're going to jump


straight into it and


we have this notebook.


Here is the data that we're going


to be using. I will load that


in in a moment but first thing I


want to do before doing that is


import and initialise everything we


need. So


obviously when we are downloading that


data I'm going to be using requests


for that so I'm going to import requests.


And


for our actual training of the model


we're going to be using both


HuginFaces Transformers and


PyTorch. So I need to


import Transformers


and I'm going


to import a BERT tokeniser


class


and also a BERT


for Net Sentence Prediction class.


So BERT


for Net Sentence


Prediction.


And as well as that we need to import


Torch.


So once we've imported all those


we can initialise our tokeniser


and model. So tokeniser


equals BERT tokeniser


from Pre-trained.


I'm going to be using BERT based on case


for this example.


Obviously you can use another BERT model


if you'd like.


So I'm going to copy that


and initialise our


model as well.


Okay.


And we can run that.


And now let's extract


this data. So this warning


here, don't need to worry about that. That's just saying


if you are using this model


for inference you shouldn't


because you need to train it


a little bit. So


we don't need to worry about that, that's fine.


Because we are going to be training.


Now what we do need to do


is get this data.


So new data equals


requests.get


and we just take


this link. I will keep


this link in the description


for you so you can just copy it across


if you want to follow along.


And we should see that we get a 200


response there, that's good.


So all we need to do is extract the


text from that and we're going to store it


in another variable


here, text variable.


And if we go back to our


text variable and if we


just have a quick look at what we have in there


we see that we have all of these paragraphs.


This is from the


Meditations by Marcus Aurelius.


You can get that book online, that's why


I'm using it. And the language


is a bit unique as well so that's


why I want to use it because


when we're using net sentence prediction


we're training the BERT model


to better comprehend


the style of language that


we train it on. So


in here we have our


paragraphs and they're all separated by a new


line character. So I'm just going to add


another little bit of code here


which is a split by new line


character. And we have a look


here, we now have a list


containing paragraphs


and that's our training data,


that's what we want to be using.


So when


we're using NSP we


want to create a 50-50


split of sentences


that are random and sentences


that are not random.


So we're going to be taking


sentence A's and


50% of the time we're going


to be adding the genuine


sentence B for that sentence A's


e.g. the sentence that follows it in


the text. And then


the other 50% of the time we're going to be


just choosing a random sentence and


pulling that in and using


that.


So to


do that we first


want a bag of sentences


to actually pull that


text from. So


the reason we can't just use text


directly is because if


we for example look at this


we see that we have multiple sentences


in this single paragraph.


So I just split


by period


we get this one


two three four


so we get four sentences and this empty


one at the end as well which we need to


remove. So


what I'm going to do


is loop


through our text


here so the text variable


split every


sentence by a


by the period characters


and


append all of those to a


new list so a flat list containing


just sentences so no paragraphs


just sentences.


And at the same time we'll need to make sure we don't include


these empty ones because we get those


with almost I think actually


every paragraph in there.


So we need to


make sure we don't include those. Now


to create this bag we


write something like this so we want to


go through each


sentence. So we want


each sentence


from each paragraph


so sentence


sorry sentence for


each paragraph


in the text


for the sentences


so for sentence


in so this is where we're getting


sentences from so


paragraph dot split


we


split by the period.


And as


well as that we also need to add that condition


that we don't want any sentences that


look like this. So we just add


that in so if sentence


is not equal


to that


and


that should be OK. So let's check


length. OK so


we get 1372


sentences from that.


And we'll actually want to


save this to a parameter


because we're using it later.


So we


now have the 1300


sentences to sample


from and now we want to do is


loop through each sentence within


text or each paragraph within text


choose a sentence


from each paragraph if


there's multiple paragraphs only


multiple sentences only


and


then 50 percent of the time


select a random sentence


from a bag and append that to the end


50 percent of the time


append the actual genuine net sentence


onto it and then we create labels


as to whether we have randomized


it or not randomized it.


So to


create that random


50 percent we import


the random library and


we also want to


initialize our sentence


a list sentence


b list.


And we


also need to initialize a


label list. OK so


that'll be 0 or 1.


Now what we want to do


is loop through each paragraph in our


text so for paragraph


in text


and then here


we extract our sentences like


we did with the bag before so


we go sentences equals


and here we


want to write sentence


for sentence


in paragraph


dot split


and remember we have those


random empty sentences we don't want to


include those so we


write if sentence


is not equal


to that


empty sentence.


So we


have now so we're now looping


through each paragraph and we've


split each paragraph into sentences.


Now what we want to do


is check


if that paragraph e.g.


our new sentences variable has more


than one sentence. So


we'll do


number of sentences


equals the length of


sentences and


then we say if


is greater than one


oops OK don't excuse it


right now and then


we apply our 50-50 logic


and append


that to our actual training data.


Otherwise if it's just a single sentence


we don't actually add it


to the training data. I mean ideally


we would want to do something like that but


for this use case


I don't want to get


make things too complicated.


So the reason I'm doing that is for example this


sentence is just a single sentence in that


paragraph and we can't guarantee


that each continuous paragraph


is talking about the same subject


you might switch. So


for the sake of simplicity


I'm just going to ignore


the single sentence paragraphs


although we do have them in


our bag so they


can be pulled in as potential sentence


bees when we randomise


the selection.


Now what we want to do is


set the sentence that we will


start from. So we write


start equals


random


randint


so this is only if we have more than one


sentence remember.


So our


random randint so this


is going to be the start sentence


in the case that we use


sentence A and B consecutively


so we don't randomise sentence B


we want to make sure


that we have enough space


at the end of our sentences.


So here


to take both sentence A and


sentence B. So let's say


for example we have


I'm going to use an example here so we have


0, 1, 2, 3, 4


let's say this is our paragraph


we have 5 sentences in here


we want the start


sentence to sentence A


if we select 4


then we don't


have a sentence B to select from.


So what we are going to do here


is say you choose a


random integer between 0


and we want 3 to be the maximum


there.


So how do we do that?


We've got the number of


sentences here so this value will be 5


in this case so


we would say number of sentences


is 5 minus


2 because we don't


the maximum value


we want to select is 3 in this case.


So it's going to be the number


of sentences


minus 2.


Ok.


Now


what we do is we do our 50-50


randomize or not randomize


sentence B.


So if random.


random so this will just


select random float between values


0 and 1.


If that


is greater than 0.5


then let's say we'll


make this our random selection.


Ok. So for the random


selection


what we do is sentence B


.append


and then here we would


append a random sentence


from our bag up here.


So to


do that we would just write bag


and then in here we need to


select a random integer


like we did up here. Ok. So


we're going to use that same


function. So random.


randint


and that needs to be between 0


and the length of


our bag minus 1.


So we use bag


size that's why we


have it.


So bag size minus 1.


Ok. Now we'll select a random


sentence B from that bag for


us. And


as well as that we


also want to set a label.


So our label in this case


would be


a 1. So


we have the 0 which means it is the next sentence.


We have a 1 which means it is


not the next sentence. So we


set 1.


Now our sentence A


it our sentence A gets selected.


It's the same thing no


matter whether we have


the random sentence


B or the not random sentence B.


So we can actually write


our sentence A append


up here.


And this is just


going to be sentences.


And in the


index we have start which is


our value from here.


Ok. So we


have the random option.


Now let's


do our not random option.


So in here we


write sentence B append.


And


this needs to append sentences


start plus


1. So the following sentence


after our sentence A.


And our label here


would be 0


which means it is the next sentence.


So let's


there's quite a lot of code. Let's run


that and see what we get.


Ok.


Now


what I want to do


is let's have a look


at the first


few labels. See if we have a


mix of different ones in there.


Ok. We just have 1, 1, 1. So I'm going to


rerun this because I want to show you the


difference between 0's and 1's here.


Ok. So we


have these. So let me


print out


what we have.


So


for i in


range 3. So I'm just


doing this so we can print and see what


we actually have in our training data.


So I want to


print the


label at that


index. And then


I want to print the


sentence A.


At that


index. And we'll follow that


with


a new line character


and a few dashes so we can


distinguish between the start and end of


sentence A and B. And then we will


do print sentence


B.


And then I'm just going to add in a new line there


to distinguish it from the next set of


answers.


So


see here that we have


0. We have


our sentence A.


And our sentence


B is a continuation


of that first sentence.


Because we have that label


0. We know that.


So we have sentence A


here. And again


this one here is a continuation


of this sentence


A. And then down


here we have a 1. So this is where we've


selected a random sentence


B.


And if we


read this. I know it's not the easiest


thing to read.


Yeah the difference. There's


reasonably clear


difference in the context there.


Okay. Now


this won't always work. In some


cases we might select even


the same sentence for sentence A and B.


But for what


we're doing here I think this is a completely


reasonable


way of


going about it. Because we don't want to


over complicate things. If we wanted


to really be very strict


on it we could add in some extra


logic which confirms that we


are not getting a


sentence B from around the same area as


sentence A for example. But for


now this is I think fine.


Okay.


So we've now prepared our


data. What we need to do now


is tokenize it. So


to tokenize our


data we're just going to use a


tokenizer which we've already


initialized. And in here


we can actually just pass our sentence


A and sentence B


like this. And our tokenizer


will deal with how to fit both of those together


for us. So that's


pretty useful. We're going to


be using PyTorch. So we want to return


tensors Pt.


And as


well as that we need to


truncate or pad each


one of those sequences to


a maximum length


of


And we truncate


using this.


And we also set


padding equal to max


length.


Okay. So


that should be


okay. Let's have a look at what we


have. We see that we have input


IDs, token type IDs and attention


mass. Let's have a look at


what they look like. So you see


here we have all these different


vectors and that is a single


pair of sentence A


and sentence B.


And we have quite a few


of those.


Now our token


type IDs


what we would expect is sentence


A would have a token type


ID of 0 and


sentence B would have token


type ID of 1.


We don't see those ones in there.


So let's expand


that out a little bit.


So we'll go with


token type


IDs.


Let's go with number 0.


Okay. So now


we see okay the reason is because


they're in the middle here.


So what we're seeing here is sentence A


followed by sentence


B. And then


these remaining 0 tokens


are our padding tokens.


So we can also see that if we switch


across to input IDs


we see that we have


all these padding tokens.


And as well another item


that the tokenizer


does for us automatically is


adds a separator token in the middle


of our sentence A and B. So


sentence A is this, sentence B


is this.


Okay.


So


we have our


input tensors. We also need to


build our labels tensor.


And to do that


we just we add it to this


inputs variable.


So we have inputs labels.


And we set


that equal to torch


long tensor.


And


this is a little bit


different. So let me


just expand that out.


So let's say we just add labels


in here.


So sorry


label.


And we just get this one big tensor


which is not really in the


correct format that we need.


We need each one of these


to


match to our


input IDs, token type


IDs and attention mask. So


what I mean by that is


if we just


have a look at this input IDs


you see that we get


it's like a list within a list.


We need that but for our


labels as well. They're in a different format


at the moment as you can see.


So we could try


transposing that.


But you see that doesn't actually do anything


because it's just a single


invention. So it's just switching everything


around.


So let's remove that transpose


and let's add a list inside here.


You see now


we're getting somewhere


not quite there yet. So now we have


a list within a list.


And now what we do


is we transpose it and now


we get what we need. So we have this


almost vector of


each of these and each one of these here.


So this vector


matches up to this


value here.


And this one


matches up to this one. And that's


what we want. So let's copy


that and put it here.


So now


we have all the sensors


we need for training our model. And


what we now need to do is set up the input


pipeline for training.


So when we're training we're


going to need to use a PyTorch


data loader object.


And to create that data loader


object we need to create


a PyTorch data set object


from our data.


So to do that we


write this. So we're going to be using


a data set


class here. So I'm going to call it


MeditationsDataSet.


And in here we write torch


utils data


data set. So that


makes sure that we are


using the correct format for the


class for a data set.


Now we need to define a few


methods inside here.


So our initialization method


and for our initialization


method we need to be able to pass our


data. So we'll pass


it through this encodings variable.


And all we need to


do in here is assign our encodings


variable to be an internal attribute


of that data set or that


class. So write self


encodings


equals encodings.


So that allows us to


create our data set class.


And then our


data loader needs two


different methods from this class as well.


It needs a get item method


and a


length method. So let's do


the length method first. It's easier.


So our length


we don't need to pass anything to this.


It's just the same


as when you would write


this and you put something


inside it. So list


0 1. We get that length.


That's exactly what we're doing here.


So this creates a


enables you to do this same


method on your class.


And


inside here all we need to do is


return the length. So what


length should we return? Well


if we just do length


of inputs


we get 4 because we only


have 4 items in there.


So we don't want that.


We actually want the number of samples


that we have within


our inputs. So what


we can do instead is we write inputs


input ids


shape 0.


So we have these


317 items.


So just show


you here. See this is


our encoding


size. So the


max length we set here.


And this is the number of


sentences or sentence pairs


that we have.


So


we take that


and we return it. But obviously we


don't have inputs. We now have this


self encodings. So


swap it like


that. And then


we want to


pass this get item


method. And what


this does is given a certain


index it will return


your 4 dictionaries


your input ids, token type ids


attention mask and labels


dictionary which we've created down here


for that specific index.


So we need to


allow it to take an index


argument there.


What we do


is return


let me


show you down here what that would look like.


So we want


to create a dictionary just like


we have up here but we just want that for that


specific index.


So what we write


is key.


And then we write our value


index. So


maybe it makes more


sense for me to write so tensor.


So our key is our


input ids, attention mask


and so on. Our tensor is obviously the tensor


inside there but we have the full tensor


containing all 317


items.


So then we pull out the


index for


that tensor. But we need to make sure we're


doing that for each of our


items. So


because we have multiple tensors


here don't we? We have the 4


input ids, labels and so on.


So


we do for


key


and tensor in


and in here we would write


let's say we do


inputs items.


let me just


take that out so you can see.


So that gives us our dictionary items.


And if we


do a for loop so


for key


tensor in


we want to


print


the key.


So you see that we're looping


through each one of those.


And we also get the tensor out for each


one of those as well.


But we're specifying a


certain tensor


with each one. So let's say


we want 0 here.


We get the


0 tensor and nothing


more.


But we want to


specify an index so we copy


that.


And it's what we're going to return here.


Except here


we change it to self


encodings.


So that's our class.


And


with that we can


initialize our data set object.


So data set equals


meditations data set.


And then all we need to do


is pass our


data which is


just inputs here.


Like that.


OK. So that's our data


set ready. Now we can initialize


our data loader.


And we do that


like this. So we


do loader


torch


utils


data dot data loader.


We pass our data set object.


We also want


to specify the batch size.


So I'm going to use batches of


And then we also want


to shuffle our data set


as well. So we write shuffle


equals


true.


And that's our data


loader. So


now


now we just need to set


up a few model


training parameters. So


the first thing we want to do


is move our model


to a GPU if we have


a GPU. So


to figure that out


what we do is write


let me do this


torch


device CUDA.


So this is we say


we want to use a CUDA enabled GPU


if torch


dot CUDA


is available.


So this will check our


environment and check if we have


a CUDA enabled GPU.


If it isn't available


we want to use a


torch device


CPU.


So run that and see


for me I have a CUDA enabled GPU


so it comes up with this.


So we saw that in


device. And then


what we can do is move


our model and also move our tensors


later on to that


device for training.


So we just write model


to device.


And we'll get


a lot of output from that.


Just ignore that we don't need to worry about


it.


And we can also


activate our models training mode


like that.


OK. So


we've moved our model over to


GPU activate training mode.


Now what we need to do


is initialize our optimizer.


So we're going to be using


Adam with weighted decay for our


optimizer. So


to use that we need to import


it from transformers.


So from transformers


import


AdamW.


And we


initialize the


optimizer like


this. So we


AdamW we pass our model


parameters. And we


also want to pass the learning rate


which is going to be


5e to the minus 5.


5e to the minus


5. OK that's a


pretty common one for training transformers.


And


that


looks pretty good to me.


So now we can begin


our training loop. So


first I want to


import something called TQDM.


Now this is purely


for aesthetics. We don't need it


for training. This is so


that we see a little progress bar during


training otherwise we don't see anything.


So I just want to include


that so we can actually see what is going on.


So from TQDM


import TQDM. So this is optional


we don't need to include it. It's up to you.


But I would recommend it.


We'll train


for


let's go with 2


epochs.


Again we don't want to train transformers


too much because they will easily


over fit. And to be honest they'll probably


over fit on this dataset because it's very


small.


But that's fine. We just want


to use this as an example.


So we're going


to train for 2 epochs.


And because we're using TQDM


we want to set up our


training loop like this. So we wrap


it within a TQDM


instance.


And all we do here is pass our


data loader. So we


create that up here.


That's our PyTorch data loader.


And we also


want to write leave


equals true.


So this is so that we can see


the


progress bar.


And then we


loop through each batch


that will be generated


by that loop generator.


So for batch in loop.


So now we're in our training loop.


What we want to do here.


Very first thing


is set our


optimizer's gradients to 0.


So obviously in the


very first


loop that it's fine.


It doesn't matter. But every loop after


that our optimizer


will have a set of gradients that have been


calculated from the previous loop. And we need to


reset those.


So we write optim


and


after that


we can load


in our


batches or our tensors from


our


batch here.


So we want input


IDs equals


a batch.


And we access it like a dictionary.


So we have input IDs.


So input IDs.


And


one other thing that we need to do is our model


is on our GPU. So we need to move


the data that


we're training on to our GPU


as well.


So we just write that.


OK.


And copy


this. So we have


one more.


So we have all


these that we create


up here. So input ID, setting type ID


attention mask and labels. We want


all of those.


So token type IDs


attention


mask


and


labels.


OK. So


initialize our gradients.


We have


pulled in


our tensors and now we can


process them


through our model. So we do model


input IDs.


We have token type


IDs.


We also have the


attention mask.


And we


also have our labels.


OK.


So that


will create


two tensors for us in the outputs.


It will create a logits


tensor which is our prediction.


And it will create a


loss tensor which is the


difference between


our prediction and our labels.


So let's extract that


loss. So we do outputs.loss.


And then we also


after extracting that loss we need to calculate


that this is the overall loss. We need


to calculate the loss for every parameter within


our model so we can optimize on that.


So we just write loss


backward.


Yeah backward.


And then


we do optim step.


And this will


use our optimizer


and take a step to


optimize based on the loss


calculated here.


And that is all we actually need


for our training loop.


We do also have the TQDM up here


as well. So I just want to


use that.


And what we're going to do is we're just going to set


the description of our loop


at this current step


equal to the epoch.


So this is just


purely aesthetics. We don't


need this for training but it's just so we can


see what is going on.


And we also want to loop set


postfix. And here I'm going to


add in


our loss which


is just going to be


loss equals loss.item


like that.


Now that should be okay.


Let's give it a go.


See what happens.


Okay.


So


that looks pretty good.


So you can see that our model is training. Loss


is reducing.


Now there isn't that much training data


so we're not going to see anything crazy here


but we can see that


it is moving in the right direction. So that's pretty good.


So that's


everything for this video.


It's a pretty long video.


Recorded for 41 minutes.


It'll probably be a little bit short for you.


But yeah that's long.


that's everything for this video.


I hope it's been useful.


And I will see you in the


Build NLP Pipelines with HuggingFace Datasets

Hi, welcome to this video. We're going to have a look at Hugging Faces data sets library.


We're going to have a look at some of what I think are the most useful data sets.


And we're going to look at how we can use the library to build


what I think are very good pipelines or data input pipelines for NLP. So let's get started.


So the first thing we want to do is actually, well, install data sets. So we'll go


clip install data sets and that will install the library for you.


After this, we'll want to go ahead and import data sets.


And then we can start having a look at which data sets are available to us.


Now there's two ways that you can have a look at all of the data sets.


The first one is using the data sets viewer, which you can find on Google.


You just type in data sets viewer and it's just an interactive


app which allows you to go through and have a look at the different data sets.


Now I'm not going to, I've already spoken about that a lot before and it's super easy to use.


So we're not going to go through it. Instead, we're just going to have a look at how we can


have view everything in Python, which is the second option.


So first we can do this. So we just list all of our data sets.


Now I'm going to just write dslists here.


And from this, we will just get, I think it's something like 1400 data sets now.


So it's quite a lot. So if we go len of all dslists.


So yeah, it's 1.4 thousand data sets, which is obviously a lot.


And some of these are massive as well.


So if we, for example, if we were to look at the Oscar data set,


so in dslists, we could go data set for data set in dslists.


If Oscar is in the data set. So these are just data set names.


Okay, and we have Oscar, I think PT is, what is PT?


I imagine it's probably Portuguese. And then we have all these other ones as well.


But these are just, these are users uploaded Oscar data sets.


This is the actual Oscar data set that's been sold by Hugging Face and it's huge.


It contains, I think, more than 160 languages.


And some of them, for example, English, obviously English is one of the biggest ones,


that contains 1.2 terabytes of data.


So there's a lot of data in there, but that's just unstructured text.


What I want to have a look at is the squad data sets.


Squad data sets. So we're going to be using, we're just going to use the original squad in this video.


But you can see that we have a few different ones here.


So Italian, Spanish, Korean, you have Thai, Thai QA squad here,


and then also French as well at the bottom.


So you have plenty of choice.


Now, obviously, you kind of need to know what sort of data set you're looking for.


I know I'm looking for a squad data set.


So I've gone, I've looked for squad.


There are other ones as well, actually.


If I change this to lower, you'll see those also pop up.


Okay, so we have like this one here and this one.


This one doesn't seem to work.


It's fine.


Now, to load one of those data sets, obviously we're going to be using squad.


We write data set equals data sets dot load data set.


And then in here, we just write our data set name, so squad.


Now, there's two ways to download your data.


So if we do this, this is the default method.


We are going to download and cache the whole data set in memory.


Which for squad is fine.


I think squad, it's not a huge data set, so it's not really a problem.


But when you think, okay, if we wanted the English OSCA data set, that's massive.


That's 1.2 terabytes.


So in those cases, you probably don't want to download it all onto your machine.


So what you can do instead is you set streaming equal to true.


And when streaming is equal to true, you do need to make some changes to your code,


which I'll show you.


And there are also some things, particularly filtering, which we will cover later on,


which we can't do with streaming.


But we will just go ahead and for now we're going to use streaming.


We'll switch over to not streaming later on.


And this creates like a iteratable data set object.


And it means that whenever we are calling a specific record within that data set,


it is only going to download or store that single record or multiple records in our memory at once.


So we're not downloading the data set.


We're just processing it as we get, which is, I think, very important.


And it is, I think, very useful.


Now, you can see here we have two actual subsets within our data.


If we want to select a specific subset, all we have to do is rewrite data sets again.


So let me actually copy this.


So we copy that.


And if we just want a subset, we write split.


And in this case, it would be train or validation.


And if I just call execute that.


So I'm not going to store that in our data set variable here,


because I don't want to use just train.


We have this single iterable data set object.


So we're just pulling in this single part of it or single subset.


And we can also view.


So here we can see we have train and validation.


If you want to see it in a more clear way, you can use dictionary syntax.


So sorry, data set keys.


You can use dictionary syntax for most of this.


So we have train and validation.


Now there's also, so the moment we have our data set,


we don't really know anything about it.


So we have this train subset.


And let's say I want to understand what is in there.


So what I can do to start is I write a data set train.


And I can write, for example, the data set size.


So how big is it?


Right, data set size, data set, not data size, size.


Don't know what I was doing there.


Let me see that we get, so it's like, so 80, about 90, 90 megabytes.


So reasonably big, but it's not anything huge, nothing crazy.


We can also, so we have that.


We can also get, if I copy this, you can also get a description.


Let me see what the data set is.


So SQUAD, I didn't even mention it already,


but SQUAD is the Stanford Question Answering Data Set.


So use it generally for training Q&A models or testing Q&A models.


And you can pause and read that if you want to.


And then another thing that is pretty important is


what are the features that we have inside here?


Now we can also just print out one of the samples,


but it's useful to know, I think.


And this also gives you data types, so it's kind of useful.


So we have ID, title, context, question, and answers.


All of them are strings.


Answers is actually, so within answers we have, it says,


Sequency, we can view it as a dictionary.


But we have a text, a attribute, and also an answer star attribute.


So that's pretty useful to know, I think.


And to view one of our samples,


so we have all the features here,


but let's say we just want to see what it actually looks like.


We can write data set and we go train.


And when we have streaming set to false, we can write this.


But because we have streaming set to true, we can't do this.


So instead what we have to do is we


actually just iterate through the data set.


So we just go for sample in data set.


And we just want to print a single sample.


And then I don't want to print anymore,


so I'm going to write break after that.


So we just print one of those samples.


And then we see, okay, we have the ID, we have title.


So each of these samples is being pulled from a different Wikipedia,


pulled from a different Wikipedia page.


In this case, the title is a titled page.


So this one is from the University of Notre Dame Wikipedia page.


We have answers.


So further down, we're going to ask a question and these answers here.


So we have the text, which is the text answer.


And then we have the position,


so the character position where the answer starts within the context,


which is what you can see here.


Now we have a question here, which we're asking.


And then the model, the Q&A model is going to


extract the answer from our context there.


Okay.


So we're not going to be training model in this video or anything like that.


We're just experimenting with the data sets library.


We don't need to worry so much about that.


So the first thing I want to do is have a look at how we can modify some of the features in our data.


So with SQUAD, when we are training a model,


one of the first things we would do is we take our answer start and the text


and we will use that to get the answer end position as well.


So let's go ahead and do that.


So I first I want to just have a look, okay, for sample in the data set train,


I'm just going to print out a few of the answer features.


So we have sample answer or answers, sorry.


And I just want to print that.


So print it.


And I want to say, okay, I want to enumerate this.


So I can count how many times we're going through it.


So here I'm just viewing the data so we can actually see what we have in there.


So I want to say, if i is greater than four, just break, just stop printing answers for us.


So and then we have a few of these.


So we have text and we have answer start.


We want to add answer end.


And the way that we do that is pretty straightforward.


We just need to take the answer start and we add the length of our text to that to get the answer end.


Nothing, nothing complicated there.


So what we're going to do here is modify the answers feature.


And the best way or I think the least the most common way of modifying features


or adding new features as well is to use the map method.


So we go data set.


So it's going to output a new data set.


So we write data set train equals data set train.


And we're going to use the map method.


And with map we use lambda.


So we write lambda x.


So in here we're building a lambda function.


And what we need to do.


So this is one of the things that changes depending on whether you're using streaming or not.


So with streaming equals true in here we need to specify every single feature.


So what I mean by that is let me do it for streaming faults initially.


So when streaming is false we would just write answers.


And we would write the modification to that feature.


So in this case we are taking the current answers.


So it would be x answers.


And we would be merging that with a new dictionary item which is going to be answers end.


So answer start.


So answer end is equal to.


And here what we have to do is we go x answers.


So this is a little bit messy now.


But it's just how it is.


So we're within answers and we want to take the answer start position.


And we want to add.


Let me start a new line here.


And we want to add the length of answers text.


OK so all we're doing there is we're taking answer start and we're adding answer text.


Or the length of answer text to that to get our answer end.


Now this is all we would have to write if we were using streaming equals false.


But we're not.


With streaming equals true we need to add every other feature in there as well.


I'm not sure why this is the case.


But it is.


So we need to just add those in as well.


So all they are is a direct mapping from the old version to the new data set.


So we don't need to really do anything there.


We just need to add ID.


We want to map that to ID and do that for the other features as well.


So we have also have context.


Which is exit context.


We have answer already done of course.


A question which is going to be exit question.


So ID context question answers.


Is there anything else I'm missing?


ID oh title of course.


Title just title.


Yeah so also add title in there as well.


Okay and with that we should be ready to go.


So let's map that.


And what we'll find is when we're using streaming keywords equals true.


The actual process is or the transformation that we just built is lazily loaded.


So we haven't actually just done anything there.


All we've said is we've passed this instruction to transform the data set in this way.


But it hasn't actually transformed anything yet.


It only performs this transformation when we call the data set.


So if we did this again.


This would call the data set and it would force the code to run this instruction or this transformation.


So let's run that.


And you see we actually do get an error here.


And why is that?


So let me come down.


We have so what am I doing?


And start plus delete those answers.


What's wrong with that?


Ah okay so if we look up here.


We have these items here that are within the list.


So we actually need to access that first item.


But that's good because we saw that when we first execute this code nothing happened.


And it only actually came across that error when we called a data set.


Because that's when this transformation is actually performed.


And now what we have to do is because we've already added this instruction to our data set


you know transformation or building process.


We actually need to reinitialize our data set.


So we will come back up here.


So where are you here?


So date no not that one this one.


So we need to load that again to reinitialize the all of the instructions that we've added in there.


And then we can go ahead rerun this.


And now it should work hopefully let's see.


There we go.


So now if we have a look at this.


And this is something I probably should have done but I completely forgot to.


So I should have added this as maybe a list rather than just the number.


But it's fine because you know if you come across and you need to do this.


You may want to add that in.


But we're not doing anything other than playing around with the data sets library.


So it's not really a problem.


But you can see that we have added answers and into there now which is what we wanted to do.


And also importantly is if I let me copy this bring down here.


We'll notice that we do still have all of our data sets.


So if I go here.


I don't really need to remove that's fine.


I'll just break straight away.


That's fine.


So sample sorry here.


So you see the whole thing.


And we see that we still have the ID.


We have the text.


We have the context.


We have everything in there.


Now I'm just going to show you you know why this breaks.


Or what happens if I remove these.


Okay so let me rerun that and this as well.


So yeah so this should look the same.


Do we have yet that's fine.


But then if I run this.


So before this had all the all the features but now we only have the single feature that we specified in this formula.


So the answers.


So that's why you need to when shuffle is set to true.


That's why you need to add every single feature in there.


Otherwise it's just going to remove them when you perform the map operation.


But that's only the case when shuffle is actually set to true.


Shuffle? Why am I saying shuffle? Streaming is set to true.


So let me bring this down here.


And let me also copy our initial loading code.


So here.


Because we're going to need to reload our data set now anyway.


Because we just removed all the features from it.


Okay and what I'm going to do now is just set streaming to false.


And I'm going to run this same code where we still don't have our ids or anything like that in there.


And we'll see what happens.


As well we'll also notice we'll get a loading bar here.


And it's going to take a little bit of time to process this.


Although actually with this it's probably going to be super fast.


So probably ignore that.


But it will you see okay it's taking a little bit of time.


So now it's going through a whole data set.


We haven't called the date set.


But we have used this map function.


When streaming is set to false the data set isn't lazily loaded.


And so the operation is going to be a bit more complicated.


So the operation the map operation is performed as soon as you call it.


So it's a slightly different behavior.


And the other behavior which is different is the fact that


we only need to specify the answers feature here.


So we only when we have streaming set to false.


We don't need to include every feature within the map operation.


We only need to include the feature that we are modifying or creating.


Which you know it's weird I don't know why there's a behavior difference


when streaming is true or false.


But it is there.


So if I now take this again.


Come down here and run that.


We see now that we have all of our features again.


Right so before when streaming was true.


If I run this code it would have only included our answers.


The id, title, context, question they all would have been removed.


But now with streaming equal to false they're still there.


So weird a weird.


So it's a weird feature or a weird behavior.


But it's how it is and we obviously just need to deal with it.


Now the next thing I want to show you is how we can also add batching


to our mapping process.


So typically with well pretty much every or any as far as I can think of any


NLP tasks we're going to want to tokenize our text.


So we're going to go ahead and do that for Q&A.


So we would import transformers or from transformers import a BERT tokenizer.


Let's say and I would initialize that.


So this is you know what we typically do.


We do tokenizer equals BERT tokenizer from pre-trained.


And let's say BERT base un-gased.


Okay I'll initialize that.


And then what I want to do is I'm going to tokenize my context or question and context.


In the format that SQUAD would usually expect when you're doing Q&A or building a Q&A model.


And I want to do that using the map function.


So you can do this in both streaming and non-streaming by the way.


So we just write date set.


It was train so same as before.


Date set it was train or date set train dot map.


We are using a lambda function so lambda x.


And in here we just want to say tokenizer.


So I'm not doing the usually when you write this you would include a dictionary here.


But the tokenizer the output from the tokenizer is already in dictionary format.


So we don't need to I don't need to do it in this case.


But basically what we have here is is still a dictionary.


And what I want to do is so with Q&A in your tokenizer you pass two text inputs.


You pass your question and you would also pass your question.


And you would also then pass your context.


And as usual we would we set our max length.


So usually 512.


I would set padding equal to the max length.


And also do truncation as well.


Okay so very typical tokenization process.


Nothing there's nothing different going on here.


This is what we normally do when we tokenize our text going into a transform model.


And then we want to say okay batched equals true.


So this allows us to do everything or perform this operation in batches.


And then we can also specify our batch size.


So batch size equals let's say 32.


So now when we run this where is it where is it going?


It's here. Now when we run this the map function here is going to tokenize our question and context in batches of 32.


So let's go ahead and do that.


Okay and then you can you can see that processing there.


So I mean that's that's all we really need to do with that.


So I think that's probably it for the map method.


And we'll well I'll fast forward and we'll continue with I think a few of the methods I think are quite useful as well.


Okay so that's just finishing up now.


So we can go ahead and have a look at what we've actually produced.


So come to here and say dataset train.


So what do we have now?


We have answers like we did before but now we also have attention mask.


We have input ids and we also have token type ids.


Which are the three tensors that we usually output from the tokenizer when we do that.


So we now have those in there as well.


We can also have a look another thing as well we can now rather than looping through our dataset


because we're not using a we're not using streaming which is true we're using streaming equals false.


We can now do this and we can see okay we have attention mask and it's not going to show me everything because it's quite large.


So I'll just delete that but you can see that we have the attention mask in there.


So what I want to do is say I want to be quite pedantic and I don't like the fact that


there is the fact that we have one feature called title.


Maybe I want to say okay it should be topic because it's a topic of the context and the question.


If I want to be really pedantic and modify that I could say dataset train rename column.


And to be honest you can use it for this of course but you're probably not going to you're probably


going to use it more for when you need to rename a column to make sure it aligns to whatever the


expected inputs are for a transformer model for example.


So that's where you would use it but I'm just using this example.


So I'm going to rename the column title to topic and let's print out and dataset train again.


So down here we have title and the moment we're going to have topic.


Okay so now we have topic.


So just rename column like I said come useful not in this case but generally this is usually useful.


Now what I may want to do as well is remove certain records from this dataset.


So so far we've been printing out the here we have this which is now topic.


We have University of Notre Dame.


Maybe for whatever reason we don't want to include those topics so we can say


very similar to before we write dataset train equals dataset train again.


This time I'm going to filter so we're going to filter out records that we don't want.


And again it's very similar to the syntax we use for the map function which is the lambda.


And in here we just need to specify the condition for the samples that we do want to include


or we do want to keep.


And in this case we want to say okay wherever the topic is not equal to University


of Notre Dame.


Okay so we'll run this and we'll have a look at what we produce.


So dataset train so somehow like we have number of rows here which is just over 88,000.


And we should get a lower number now.


Now this will also go through so this remember we have shuffle.


Set to shuffle what I keep calling it shuffle.


We have streaming set to false this time.


So it's going to run through the whole dataset and perform this filtering operation.


Now whilst we're waiting for that.


Now I'll just fast forward again to when this finishes in a moment.


Okay so now we have this finished and we can now run this.


It's finished and we have before we had 88,000 rows now we have 87.3.


And we should see so let me take the dataset train


topic and I want to see let's say the first five of those.


Okay now they're all Beyonce rather than before where it was the University of Notre Dame.


So we have those and what we may want to do now is say for example we're performing inference


with Q&A with a transformer model.


We don't really need all of the features that we have here.


So we would only need the attention mask, the input ids and also the token type ids.


So what we can do now is we can remove some of those columns.


So we'll do dataset train as always dataset train again.


And we want to remove those columns so remove columns.


And we'll just remove so all of them other than the ones that we want.


So do answers context id question and topic.


Okay and then let's have a look at what we have left.


Okay and then that's it so we have those final features and these are the ones that we would


input into a transform model for training.


Now I mean there's nothing else I really want to cover.


I think that is pretty much all you need to know on Iconface datasets to get started


and start building pretty I think good input pipelines and using some of the


datasets that are available.


So we'll leave it there.


Thank you very much for watching and I will see you again in the next one.


How-to do Sentiment Analysis with Flair in Python

Hi, welcome to this video on sentiment analysis using the Flare library.


So Flare is an incredibly simple, easy to use library, which contains a load of pre-built models for NLP


that we can simply import and use to make predictions.


So it actually allows us to use some of the most powerful models out there as well.


So in this tutorial, we're going to be using the Distilbert model, which is based on a BERT, but it's a lot smaller,


but almost as powerful as BERT itself.


So we're going to go ahead and begin.


First, if you haven't already, you need to pip install Flare.


And alongside Flare, you are also going to need PyTorch.


If you haven't got PyTorch installed already, you'll need to head over to the PyTorch website.


And they give you instructions on exactly what you need to install.


So we come down to here and we can see, OK, for me, I have Windows.


I want to install using Conda, using Python and then CUDA.


So this is if you have a CUDA enabled GPU on your machine.


If you don't know what that means, you probably don't.


So in that case, just click none.


But for me, I have 10.2.


So all we need to do is copy the command underneath here.


And then we would run this in our Anaconda prompt.


I already have these installed, so I'm going to go ahead and actually begin coding.


So we're going to need to use Pandas and also Flare.


So now we have imported Flare, we can actually import a sentiment model straight away.


So all we need to do is pass our sentiment model to a variable.


Which we will call sentiment model.


And we just need to write Flare.models.textClassifier and load.


And then in here, we pass the model name that we would like to load.


And in our case, it will be the English sentiment model, which is en-sentiment.


En-sentiment.


Like so.


OK, so now we are downloading the model.


And in a moment that will have downloaded and we can begin using it.


Now, obviously, we need data.


I have downloaded some data here.


Which is a sentiment data set based on the IMDB Movery reviews.


So you can find the same data set over here.


OK, so sentiment analysis on Movery reviews data set.


So it's from Rotten Tomatoes.


You can just scroll down and we have the training data and test data here.


I'm just going to use the test data and build a test data set.


I'm just going to use the test data, but we can use either.


We're just going to be making predictions based on the phrase here.


So we need to read in our data.


So it's going to read it in as if it were a CSV file.


And we will just pass a tab as our separator,


because we are actually working with a tab separated file.


OK, so here it's actually a CSV, not CSV.


OK, so the first thing you'll notice is that we actually have duplicates of the same phrase.


That is actually just how this data set is.


It just contains the full phrase initially.


So this first entry here is the full phrase.


And then all of these following are actually parts of that phrase.


So what we can do, so let's change it so we can actually see the full phrase first.


OK, so we can't really see that much more anyway, but we can see that the full phrase


has fine.


So to remove this, we just want to drop all of the duplicates


whilst keeping the first instance of the sentence ID.


So you see each one of these, they all have the same sentence ID.


It's actually only the first one that we need.


So we just drop duplicates on this column, keeping the first entry.


OK, so we're keeping the first entry, dropping duplicates from sentence ID,


and we're just doing this operation in place.


OK, so now we can see each sample is now a unique entry.


OK, so now our data is ready.


So we need to actually first convert our text into a token as list using Flare.


So Flare does this one sentence at a time. So if we, for example, pass


Hello World into the Flare tokenizer, we will be able to see what it's actually doing.


OK, so here we can see that it split each one of these into tokens.


So we've got Hello is a token, World is a token, and then we have also split the


exclamation mark at the end there.


And you can see that Flare is telling us that there are a total of three tokens.


So we can see that Flare is telling us that there are three tokens.


And you can see that Flare is telling us that there are a total of three tokens there.