CLIP Explained | Multi-modal ML

are teaching them to speak the same vector language right so this is it's very abstract


CLIP Explained | Multi-modal ML

this this vector language is like 512 dimensional space so we can't directly understand what


CLIP Explained | Multi-modal ML

or it's very difficult for us to directly understand what is actually happening there


CLIP Explained | Multi-modal ML

but these two models do actually output patterns that are logical and and make sense and we can


CLIP Explained | Multi-modal ML

see some of this by comparing the similarity between the vectors that it outputs okay so we


CLIP Explained | Multi-modal ML

can see that the two vectors for dogs running across a frosty field both the the text vector


CLIP Explained | Multi-modal ML

and the image vector are both within a very similar vector space whereas something else like elephants


CLIP Explained | Multi-modal ML

in the Serengeti is you know whether it's text or image is not here with our our two dogs running


CLIP Explained | Multi-modal ML

across the frosty field is somewhere over over here right in a completely different space so


CLIP Explained | Multi-modal ML

what we can do with that is is calculate the similarity between these vectors and identify


CLIP Explained | Multi-modal ML

which ones are similar or not similar according to clip from this from these these meaningful


CLIP Explained | Multi-modal ML

vectors that that clip is actually outputting we are able to create a content-based image retrieval


CLIP Explained | Multi-modal ML

system okay so content-based image retrieval is basically where we um using some text or using


CLIP Explained | Multi-modal ML

maybe even another image we can search for images based on their content right and not just like


CLIP Explained | Multi-modal ML

some metatextual metadata or something that's been attached to it and with clip unlike other


CLIP Explained | Multi-modal ML

content-based image retrieval systems um clip is incredibly good at actually capturing the meaning


CLIP Explained | Multi-modal ML

across the entire image so you know for example with our our two dogs running across a frosty field


CLIP Explained | Multi-modal ML

we might also be able to describe the background of that image without mentioning that there's two


CLIP Explained | Multi-modal ML

dogs in it and if we describe in such a way that um we align pretty well with what that image


CLIP Explained | Multi-modal ML

actually is what is in that image we might actually also return the image based on that so


CLIP Explained | Multi-modal ML

we're not just focusing on one thing in the image clip allows us to focus on many things in the image


CLIP Explained | Multi-modal ML

so an example of that is within this data set i've been using here there are no images there's


CLIP Explained | Multi-modal ML

one single image of the food a hot dog okay so i tried to search that and the first image that is


CLIP Explained | Multi-modal ML

returned is a dog eating a hot dog okay so it's pretty relevant but of course there are no other


CLIP Explained | Multi-modal ML

images of hot dogs in this in this data set so the other images that are returned are quite


CLIP Explained | Multi-modal ML

interesting because in some way or another they are kind of showing a hot dog so the first one we


CLIP Explained | Multi-modal ML

have a dog looking pretty cozy in a warm room with a fire in the background then we have a dog in a


CLIP Explained | Multi-modal ML

big woolly jumper and another dog kind of like posing for the camera so weirdly enough we we


CLIP Explained | Multi-modal ML

got a load of hot dog images even though it's not really um maybe it's not exactly what we meant when


CLIP Explained | Multi-modal ML

we said hot dog but a person could understand that okay we can we can see how those that term


CLIP Explained | Multi-modal ML

and those images are related now we're not actually only restricted to text to image search


CLIP Explained | Multi-modal ML

when we encode our our data when we code text and when we code images we are actually just


CLIP Explained | Multi-modal ML

creating vectors so we can search across that space in any any direction with any combination


CLIP Explained | Multi-modal ML

of modalities so we could do a text to text search image to image search we can also do image


CLIP Explained | Multi-modal ML

to text search or we can search everything we could use some text to search for text and images


CLIP Explained | Multi-modal ML

we can kind of go in any direction use any modality we want now let's go into a little


CLIP Explained | Multi-modal ML

more detail on what the architecture of clip actually looks like so clip as i mentioned it's


CLIP Explained | Multi-modal ML

these two models now these two models are trained in parallel one of them is the the text encoder


CLIP Explained | Multi-modal ML

now it's a just a generic text encoder of 12 layers and then on the image encoder side there


CLIP Explained | Multi-modal ML

are there are two different options i've spoken about there is a vision transformer model and


CLIP Explained | Multi-modal ML

also a resnet model and they use a few different sizes for resnet as well both of these both of


CLIP Explained | Multi-modal ML

these encoder models output a single 512 dimensional vector and the way these models


CLIP Explained | Multi-modal ML

is trained is is kind of in the name of clip so clip stands for contrastive learning in pre-training


CLIP Explained | Multi-modal ML

and so the the training that is used during pre-training is is contrastive it's contrastive


CLIP Explained | Multi-modal ML

pre-training now across both nlp and computer vision large models sort of dominate the the


CLIP Explained | Multi-modal ML

state of the art and the reason for this or the idea behind this is that just by giving a large


CLIP Explained | Multi-modal ML

model a huge amount of data they can learn sort of general patterns from what they see and almost


CLIP Explained | Multi-modal ML

kind of internalize a a general rule set for the the data that it sees okay so they manage to


CLIP Explained | Multi-modal ML

recognize general patterns in their modality in language they may be able to internalize the


CLIP Explained | Multi-modal ML

grammar rules and patterns in english language for vision models that may be sort of the general


CLIP Explained | Multi-modal ML

patterns that you identify or notice in with different scenes and different objects


CLIP Explained | Multi-modal ML

now the problem with these different models the reason they don't fit together very well already


CLIP Explained | Multi-modal ML

is that they're trained separately so by default these state of the art models have no understanding


CLIP Explained | Multi-modal ML

of each other and that that's where clip is is different that's what clip has has brought to the


CLIP Explained | Multi-modal ML

table here with clip the text and image encoders are trained while considering the context of the


CLIP Explained | Multi-modal ML

other modality okay so the text encoder is trained and it considers the modality or it considers the


CLIP Explained | Multi-modal ML

the concept learned by the image encoder and the image encoder does the same for the text encoder


CLIP Explained | Multi-modal ML

and we can almost think of this as the the image and text encoders are sharing a almost indirect


CLIP Explained | Multi-modal ML

understanding of the other modality now contrastive training works by taking a image and


CLIP Explained | Multi-modal ML

text pair so for example the two dogs running across a frosted field and putting those together


CLIP Explained | Multi-modal ML

into the text encoder and image encoder and learning to encode them both as as closely as


CLIP Explained | Multi-modal ML

possible for this to work well we also need negative pairs so we need something to compare


CLIP Explained | Multi-modal ML

against this is a general rule in contrastive learning you can't just have positive pairs


CLIP Explained | Multi-modal ML

because then everything can just be kind of encoded into the same like tiny little space


CLIP Explained | Multi-modal ML

and you don't know how to separate the the pairs are dissimilar okay so we need both positive and


CLIP Explained | Multi-modal ML

negative pairs so we have a positive pair okay in order to get negative pairs we can essentially


CLIP Explained | Multi-modal ML

just take all the positive pairs in our data set and we can say okay the pair t1 and i1 we can mix


CLIP Explained | Multi-modal ML

t1 with different eyes okay so we can do t1 with i2 and i3 i4 and so on so we're basically just


CLIP Explained | Multi-modal ML

swapping the pairs and we can we can understand that there's other pairs are probably not going


CLIP Explained | Multi-modal ML

to be similar as long as our data set is relatively large occasionally maybe we will get a pair that


CLIP Explained | Multi-modal ML

are similar but as long as our data set is large enough that that doesn't happen too frequently


CLIP Explained | Multi-modal ML

it's not going to affect our training it will be sort of a negligible problem so with this idea


CLIP Explained | Multi-modal ML

we can use a loss function that will minimize the difference between positive pairs and maximize the


CLIP Explained | Multi-modal ML

difference between negative pairs and that will look something like this where we have our positive


CLIP Explained | Multi-modal ML

pairs in the diagonal of the similarity matrix and everything else is something that we the dot


CLIP Explained | Multi-modal ML

product there we need to maximize and this image that you see here is actually the pre-training for


CLIP Explained | Multi-modal ML

a single batch okay so one interesting thing to note here is if we have a small batch say we only


CLIP Explained | Multi-modal ML

have a batch size of two it's going to be very easy for our model to identify which two items are


CLIP Explained | Multi-modal ML

similar which two are not similar whereas if we have 64 in our 64 items in our batch it will be


CLIP Explained | Multi-modal ML

much harder for our model because it has to it has to find more nuanced differences between them and


CLIP Explained | Multi-modal ML

and what basically the odds of guessing randomly between those and guessing correctly are much


CLIP Explained | Multi-modal ML

smaller so a larger batch size is a good thing to to aim for in this contrastive pre-training


CLIP Explained | Multi-modal ML

approach so with that i think we we have a good idea now of how clip can be used and also you


CLIP Explained | Multi-modal ML

know how it has been trained for for this so what i really want to do now is kind of show you how you


CLIP Explained | Multi-modal ML

might be able to use it as well now we're going to be using the vision transformer version of clip


CLIP Explained | Multi-modal ML

okay so we remember i said there's a the resnet and vision transformer options for that image encoder


CLIP Explained | Multi-modal ML

we're going to use a vision transformer version and openai have released this model through the


CLIP Explained | Multi-modal ML

hooking face library so we can we can go to the hooking face library and use it directly from now


CLIP Explained | Multi-modal ML

which makes it really easy for us to actually sort of get started with it so let's go ahead and do


CLIP Explained | Multi-modal ML

that now okay so for this we will need to install a few libraries here so we have transformers


CLIP Explained | Multi-modal ML

torch and data sets so data sets we need to actually get data set so i've prepared one


CLIP Explained | Multi-modal ML

especially for this so we have this image text data set and in here we don't have there's not


CLIP Explained | Multi-modal ML

much it's just 21 images or text to image pairs and we can see what they look like so


CLIP Explained | Multi-modal ML

we have this text aeroshock of a futuristic city with a large motorway okay so i tried to just


CLIP Explained | Multi-modal ML

describe this image as as best i could and yeah that's what i got and there are like as like you


CLIP Explained | Multi-modal ML

saw just now 21 of these image text pairs in there so let's go ahead and actually prepare or download


CLIP Explained | Multi-modal ML

and sort of initialize clip for our for our use so the the model id on hooking face is this


CLIP Explained | Multi-modal ML

so if we were to go to hooking face.co we could type that in here and we have the model there


CLIP Explained | Multi-modal ML

okay so this is the model that we're using over from openai here and with this model we we use


CLIP Explained | Multi-modal ML

these two we use a processor and a model so this is the model itself this is clip right this is a


CLIP Explained | Multi-modal ML

almost like a pre-processor for both our text and also the images okay so one thing we would do here


CLIP Explained | Multi-modal ML

if we have a CUDA device available we can move our model to the CUDA device


CLIP Explained | Multi-modal ML

at the moment if you try and do this with nps so if you're on mac and you have a you have apple


CLIP Explained | Multi-modal ML

silicon there are some processors or some transformations in the clip that don't


CLIP Explained | Multi-modal ML

function on nps at the moment so i would stick with cpu we're only doing inference so it's still


CLIP Explained | Multi-modal ML

pretty fast now as i was mentioning the the processor is what handles both the text and


CLIP Explained | Multi-modal ML

image preparation that needs to happen before we feed them into the actual encoder models themselves


CLIP Explained | Multi-modal ML

that make up clip so for text we do this so this is just going to be this is going to work like a


CLIP Explained | Multi-modal ML

normal text tokenizer a normal text tokenizer for text transform models is used in order to


CLIP Explained | Multi-modal ML

translate our human readable text into transformer readable ids okay so we pass the text here we make


CLIP Explained | Multi-modal ML

sure we are saying there are no images included here because the processor if we have both


CLIP Explained | Multi-modal ML

images and text it can process them at the same time we can do that here as well but i want to


CLIP Explained | Multi-modal ML

show you it separately just to show you what they're actually doing so the padding


CLIP Explained | Multi-modal ML

we need to set that to true and that is because different different sentences can have different


CLIP Explained | Multi-modal ML

lengths okay so you have like hello world and whatever i wrote before up here so this


CLIP Explained | Multi-modal ML

aerial shot of futuristic city


CLIP Explained | Multi-modal ML

aerial shot of a city these two sentences have different lengths and a transform model needs to


CLIP Explained | Multi-modal ML

see the same length being input for all of the the text that is within this sort of single batch


CLIP Explained | Multi-modal ML

so basically what it's going to do there is add what are called padding labels so it's just going


CLIP Explained | Multi-modal ML

to add a few of these up to the length of the longest sequence within that batch of of text


CLIP Explained | Multi-modal ML

items because in here we have those 22 um 20 no sorry 21 sentences so that's all we're doing there


CLIP Explained | Multi-modal ML

i'm sure that is uh and then we are returning those as pytorch sensors and then finally just


CLIP Explained | Multi-modal ML

moving them to whichever device we're using i'm using cpu here so it's not actually necessary to


CLIP Explained | Multi-modal ML

do this but i'm doing it in case you do do the same on a cuda enabled device so from there we


CLIP Explained | Multi-modal ML

have these input ids and an attention mask okay so let's have a quick look at what what those are so


CLIP Explained | Multi-modal ML

we go into tokens and we have a look at input ids okay you see we get all these literally just


CLIP Explained | Multi-modal ML

integer values and you'll see that a lot of them have this 49407 at the end all right that is


CLIP Explained | Multi-modal ML

they're the padding tokens there okay so they they are not represented as strings but they're


CLIP Explained | Multi-modal ML

represented as these integer numbers okay and we know that they're the paying tokens because they


CLIP Explained | Multi-modal ML

they're appearing several times at the end of each sequence and none of the sequences i fed in there


CLIP Explained | Multi-modal ML

were they didn't have any similar words at the end of those okay so you can see them all here


CLIP Explained | Multi-modal ML

so we know that those are the pattern sequences we also see there's like an initialization of


CLIP Explained | Multi-modal ML

sequence token there as well and then everything in between those they are tokens that represent


CLIP Explained | Multi-modal ML

a word or a part of a word from our original text so that's the input ids the attention mask


CLIP Explained | Multi-modal ML

so that's the input ids the attention mask is you'll see so here you can see that it's just


CLIP Explained | Multi-modal ML

these ones and zeros now the ones represent real tokens okay they represent real words that were in


CLIP Explained | Multi-modal ML

our from our text inputs the zeros represent where the where our processor has added padding tokens


CLIP Explained | Multi-modal ML

so this is used for the internal mechanisms of the text transformer to know which tokens to pay


CLIP Explained | Multi-modal ML

attention to which ones to ignore because we don't want to really focus on those padding tokens because


CLIP Explained | Multi-modal ML

they're meaningless they're just there to make sure we have the same size inputs going into our


CLIP Explained | Multi-modal ML

transform model that's all that is so we can go down and after we have our tokens you know what


CLIP Explained | Multi-modal ML

we what we do is we use clip to encode all of them with this get text features okay and then


CLIP Explained | Multi-modal ML

we pass our tokens and i've got two device here i think i already i already moved them to device


CLIP Explained | Multi-modal ML

so i don't need to do that again we can actually remove that okay and okay what do we get here


CLIP Explained | Multi-modal ML

so we get 21 so 21 text inputs that makes sense 512 dimensional vectors okay so they are our text


CLIP Explained | Multi-modal ML

embeddings representing each of those those text sentences that we just gave and then one other


CLIP Explained | Multi-modal ML

thing i wanted to point out here is that we have the min and max values and they're pretty big okay


CLIP Explained | Multi-modal ML

they're clearly not normalized so this depends on what you're doing if you are if you want to


CLIP Explained | Multi-modal ML

compare these vectors you need to make sure you're not using a similarity metric that looks or that


CLIP Explained | Multi-modal ML

considers the magnitude of your vectors you need to only consider the the angle so you can do that


CLIP Explained | Multi-modal ML

with cosine similarity or the alternative is that you can normalize these vectors and then you can


CLIP Explained | Multi-modal ML

also do this with dot product similarity okay so to normalize if you wanted to use that product


CLIP Explained | Multi-modal ML

similarity now you would do this okay so here we're just detaching our text embeddings from the


CLIP Explained | Multi-modal ML

the pytorch graph moving them cpu if needed we actually don't need to do that but do it here


CLIP Explained | Multi-modal ML

anyway and convert them into a non-py array and then we calculate the value that we will normalize


CLIP Explained | Multi-modal ML

that we will normalize it each vector by okay so for each each vector we're calculating a


CLIP Explained | Multi-modal ML

number and then that number is what we're going to divide them all by here okay to to normalize that


CLIP Explained | Multi-modal ML

and then after that you can see the minimum maximum is this minus 0.15 and plus 0.53 okay so


CLIP Explained | Multi-modal ML

neither of them going over minus one or plus one now now when it comes to encoding images we we do


CLIP Explained | Multi-modal ML

the same thing or very similar thing so images are also pre-processed using the using the processor


CLIP Explained | Multi-modal ML

as we did with our text but we just use slightly different parameters to start there so the reason


CLIP Explained | Multi-modal ML

we're processing these images is that clip expects a certain size of image when when we're feeding


CLIP Explained | Multi-modal ML

images into it and expects those those image pixels to be normalized as well now rgb images


CLIP Explained | Multi-modal ML

by default the the value the pixel values and they will range from zero to 255 we need to


CLIP Explained | Multi-modal ML

normalize those and we also need to resize the images so you can see you can see that here so


CLIP Explained | Multi-modal ML

the first image it has this size it's it's a pretty big image okay this is the the width and


CLIP Explained | Multi-modal ML

the height of that image now here we're taking all the images and we're processing them make


CLIP Explained | Multi-modal ML

sure we say text is is none and that will actually only output one tensor the pixel values tensor so


CLIP Explained | Multi-modal ML

we're just going to extract that straight out there and we're also going to move it to the


CLIP Explained | Multi-modal ML

device set hardware device in this case just cpu and now let's have a look at this image


CLIP Explained | Multi-modal ML

or images now so now we can see that we have this this array or tensor with three color channels so


CLIP Explained | Multi-modal ML

this is the rgb and it has a height and width of 224 so it's been you know sort of squeezed into


CLIP Explained | Multi-modal ML

a smaller size now and we have 21 days because we fed in all of our images okay so this is how we


CLIP Explained | Multi-modal ML

use the processor and this is just resizing and normalizing our images ready for the division


CLIP Explained | Multi-modal ML

transformer encoder of clip and very similar to before before we use get text features now we're


CLIP Explained | Multi-modal ML

going to use get image features and we pass in those images like that and again as you you might


CLIP Explained | Multi-modal ML

expect those images are not normalized you see that here and as we would also expect they are


CLIP Explained | Multi-modal ML

the same dimensionality as our text embeddings so that means we can compare them but before


CLIP Explained | Multi-modal ML

comparing them of course as before we we normalize them so we should normalize them again here


CLIP Explained | Multi-modal ML

um and yep same process again and we can see that those have those have changed okay cool so


CLIP Explained | Multi-modal ML

what we now want to do is calculate the similarity between all of our image embeddings and all of our


CLIP Explained | Multi-modal ML

text embeddings so we can do that in a few different ways we have cosine similarity or


CLIP Explained | Multi-modal ML

dot product similarity the reason we can use our product similarity is because we normalize


CLIP Explained | Multi-modal ML

but i'm going to show you how to do both so that if you don't normalize you can actually just use


CLIP Explained | Multi-modal ML

a cosine similarity like we do here so cosine similarity is actually just a dot product as


CLIP Explained | Multi-modal ML

a numerator between the text embeddings and image embeddings and in the denominator we have just


CLIP Explained | Multi-modal ML

normalized the norm values of both of those okay that is that's all it is actually so it's it's


CLIP Explained | Multi-modal ML

pretty pretty simple and if we plot those similarity scores between those we get this


CLIP Explained | Multi-modal ML

so we would expect along this diagonal here we'd expect these to be the highest similarity values


CLIP Explained | Multi-modal ML

which say represent the the true pairs okay between the images and the text now we have some


CLIP Explained | Multi-modal ML

that are not quite there like here and there is this image text pair which is more similar even


CLIP Explained | Multi-modal ML

with with something else you know i very quickly put these together so they're not always going to


CLIP Explained | Multi-modal ML

be perfect so we have one here that is maybe not perfect um but again there is also a lot of overlap


CLIP Explained | Multi-modal ML

between these images so there are several images of city skylines and a lot of time i described


CLIP Explained | Multi-modal ML

those as futuristic cities in you know in whatever with a big motorway or something on those sides


CLIP Explained | Multi-modal ML

so that's probably where we're getting that from um now if we were to calculate the dot product


CLIP Explained | Multi-modal ML

similarity between these we would expect it to be the same okay now um okay from this this calculation


CLIP Explained | Multi-modal ML

dot product similarity we do seem to get a very similar um set of similarities at the end there


CLIP Explained | Multi-modal ML

but are they the same well if we go down here we can see that they pretty much are so we can't do


CLIP Explained | Multi-modal ML

a straight comparison we can't do we can't do cosine similarity equals dot similarity because


CLIP Explained | Multi-modal ML

the numbers are actually slightly different but they're only slightly different because there's


CLIP Explained | Multi-modal ML

sort of a floating point error because these are all floating point numbers so we get very very


CLIP Explained | Multi-modal ML

very small um differences between the numbers and you can see that here so taken we've calculated


CLIP Explained | Multi-modal ML

the difference between them between the numbers and the two arrays and then we're looking okay


CLIP Explained | Multi-modal ML

what's the minimum difference between them it's zero okay that's what we'd expect that's where


CLIP Explained | Multi-modal ML

the numbers are exactly the same what's the maximum difference between the numbers and it's


CLIP Explained | Multi-modal ML

2.98 to the minus eight so like 0.0000000 and so on two so very small number and okay with that we


CLIP Explained | Multi-modal ML

know it's just floating point errors between those two similarity arrays so that's pretty cool to to


CLIP Explained | Multi-modal ML

see that and we can use this exact concept of comparing with with a cosine similarity or dot


CLIP Explained | Multi-modal ML

product similarity to actually search through all of our images with like a text prompt for example


CLIP Explained | Multi-modal ML

but that's not all that clip is good for clip is also has this amazing performance as a zero


CLIP Explained | Multi-modal ML

shot model for different tasks okay so not even just one task but actually different tasks so


CLIP Explained | Multi-modal ML

um it is it performs incredibly well out of the box for classification and we'll go through this


CLIP Explained | Multi-modal ML

in more detail in a future video but the idea is that given a set of um classes from a classification


CLIP Explained | Multi-modal ML

image classification data set you can maybe you can modify them a little bit the class names to


CLIP Explained | Multi-modal ML

make them more like a sentence and then you use this same idea of comparing all of your your your


CLIP Explained | Multi-modal ML

text representations of the classes with a set of images from the data set and with this you just


CLIP Explained | Multi-modal ML

calculate the similarity between those and the the text embedding or the you can think of it as a


CLIP Explained | Multi-modal ML

class embedding that gets the highest similarity to your image is the predicted class okay so you


CLIP Explained | Multi-modal ML

have zero shot classification like that super easy another use case is object detection so


CLIP Explained | Multi-modal ML

you let's say you have maybe you're looking for a cat or a butterfly in an image okay and you okay


CLIP Explained | Multi-modal ML

when you're looking for the cat you're gonna you're gonna use a chunk of text that says um


CLIP Explained | Multi-modal ML

a fluffy cat okay and you encode that with clip and you get your text embedding and then what you


CLIP Explained | Multi-modal ML

do is you break up your image into all these little patches and you just slide through all


CLIP Explained | Multi-modal ML

of those patches okay you can you can include like an overlap so you're going over those over


CLIP Explained | Multi-modal ML

like you're not missing anything between patches so you're just sliding through your image and


CLIP Explained | Multi-modal ML

with each part of the image that you slide through you extract the image from that you process it


CLIP Explained | Multi-modal ML

through clip and then you compare the encoding for that image against the tips that you create so a


CLIP Explained | Multi-modal ML

fluffy cat and what we will see is that patches of the image that contain what it what it is you've


CLIP Explained | Multi-modal ML

just described will have a higher similarity rating okay and then you can overlay those scores


CLIP Explained | Multi-modal ML

back onto your image and you will find that the that clip is able to essentially identify where


CLIP Explained | Multi-modal ML

in your image a specific object is and you are just describing that image using a natural language


CLIP Explained | Multi-modal ML

prompt now these are only a few use cases of clip and only really scratch the surface of what is


CLIP Explained | Multi-modal ML

actually possible with this model we also see being used in and like i said the diffusion models


CLIP Explained | Multi-modal ML

like dali which is a great example of how powerful clip can actually be so that's it for this


CLIP Explained | Multi-modal ML

that's it for this introduction to clip i hope it's been useful as i said we're going to go


CLIP Explained | Multi-modal ML

into more detail on the different use cases of clip and how to apply clip for these use cases


CLIP Explained | Multi-modal ML

in future videos but until then that's it for now so thank you very much for watching


CLIP Explained | Multi-modal ML


How to build a Q&A AI in Python (Open-domain Question-Answering)

Today we're going to have a look at Open Domain Question Answering


and how we can fine tune our own Retriever model


to use for Open Domain Question Answering.


We're going to start with a few examples.


Over here we have Google and we can ask Google questions


like we would a normal person.


So we can say, how do I tie my shoelaces?


So what we have right here is three components to the question and answer.


And I want you to remember these


because these are relevant for what we are going to be building.


We have the query at the top.


We have what we can refer to as a context, which is the video,


which is where we're getting this small, more specific answer from.


And we can ask another question.


Is Google SkyNet?


So we have our question at the top.


We have this paragraph, which is our context.


And then we have the answer, which is yes, which is highlighted here.


So it's slightly different to the previous one where we had the video.


This time we have actual text, which is our context.


And this is more aligned with what we will see throughout this video as well.


Now, what we really want to be asking here is one, how does Google do that?


And more importantly, is why should we care?


Now, we can imagine this problem as us being in a really big warehouse.


Now, we don't know anything about this warehouse.


We just know that we want a certain object.


Now, we don't know the specific name for the object.


We just kind of know what it looks like.


Now, in this warehouse, everything we assume is probably going to be organized in some way.


So the first thing we're going to do is have a look at the products around us


and try and figure out some sort of order.


And once we have figured out how this warehouse is structured


and how we can search through the warehouse, we need to figure out,


okay, everything is maybe organized based on the name of the product.


We don't know the name of the product.


We're kind of going to struggle along.


We're going to spend a lot of time searching through a lot of items


in order to find what we actually want to find.


That is like a normal traditional search experience


where you have to know the exact keywords that you will find


with whatever it is you're looking for.


You have to know the product name.


Now, this sort of natural language search that we just saw with Google


is not like that warehouse where you're just alone trying to figure out


how it's structured and how to search through it.


Instead, it's almost like you have a guide with you,


someone who knows this warehouse.


They basically live in this warehouse.


They know where everything is.


And if they don't know where a specific object is,


they can point you in a pretty good direction


and really help you find everything probably a lot faster.


Because as well as knowing where everything is,


this person also speaks the same way that we do.


We can ask them a question like,


okay, guide, where are those marble-like things?


And with that, they will hopefully be able to guide you in the right direction.


And maybe they'll be able to guide you right to that product


or at least in this area that they are found in.


So that's the difference between traditional search


and a question and answering search.


Now, don't get me wrong.


There are places where you do want to keep traditional search.


But particularly for unstructured text data or as we saw earlier,


video and audio data, this sort of Q&A approach can be really powerful.


So that leads us on to the second question or first question I asked,


which is how does Google do that?


Now, Google is pretty complex.


But at the core of what we saw there was something called


Open Domain Question and Answering or ODQA.


Now, ODQA is a set of language models and technologies


all piled together into a open domain question answering pipeline.


Now, that pipeline at its simplest will look something like this.


So we have our, let's say at the top, we have our question.


That question is going to come down here and it's going to hit


what is a retrieval model, which is what we will train or fine tune in this video.


Now, this retrieval model, that will handle taking our query


and converting it into a vector.


Now, we convert it into a vector because then we compare it to other chunks of text


and we can encode the semantics and meaning behind that text


rather than just the keywords.


So that's why we can search for concepts and meaning


rather than just keywords, as I mentioned earlier.


So we have this retrieval model.


Now, the retrieval model creates a vector, but then we need something that allows us to search.


So we need other vectors to compare to.


Now, where do we store those vectors?


Well, we have a vector database.


Now, a vector database, let's bring that over here.


This is going to contain context vectors.


So you remember earlier on with those Google searches, we had the question,


we had the context and the answer.


This is where that is relevant.


So in here, we have loads of context.


So just put C, but they've all been converted into vectors


using the same model we have up here, the retrieval model.


We just did it before we started searching.


So we index those contexts into our vector database.


Now, at search time, we convert our question into a vector.


That comes down into the database and the vector database will compare


that question vector to all of these context vectors


and it will return the ones that are most similar.


So maybe we want to return the top five most similar contexts.


At this point, if we are just using a retrieval model and a vector database,


we can return those.


We had our question and we can return these contexts to the user.


So that would be like in our earlier examples, we ask a question


and Google returns the page or it just returns a paragraph to you


rather than highlighting the specific answer.


And these are the components that we're going to cover today.


And in a future video and article, what we are also going to include


is a reader model.


So a reader model is added onto this Open Domain Q&A stack


or it's the last component of the stack.


And what this does is it takes each of your context vectors


and it reads them so it has a look at the context vectors


and we have this long text here and it says,


okay, given the question, which we also feed into our reader model,


I think the answer to that question is right here.


So it allows us to extract a very specific answer from a longer chunk of text, the context.


So that's the Open Domain Q&A structure or pipeline.


And now I think we should move on to actually fine tuning the retrieval component of that pipeline.


The first thing we need to think about when fine tuning our retrieval model


is whether or not it actually needs to be fine tuned


because there are retrieval models out there already that we can just download and use.


And there's this really good concept that I saw from in one of Nils Reimer's YouTube videos


where he's going to talk and he talks about the long tail of semantic relatedness.


And the basic gist of it is that you have common knowledge that everyone,


pretty much everyone knows about and you have a lot of data and benchmarks in that area.


So that would be our cat versus dog example up here.


So imagine you're on the street, you walk up to a stranger, you ask them a question,


what's the difference between cat and dog? They're probably going to know the answer.


Everyone knows that. Then you get more specific.


So you ask them, OK, what's the difference between C and Java?


Some people know, some people will not. And then we get even more specific.


PyTorch versus TensorFlow. And then we get more specific.


Roberta versus Dberta and then TSA versus Mirabert.


As we get more specific, less and less people know what you're talking about.


And with that, there are less data sets and there are less benchmarks.


But at the same time, that's where most of the interesting use cases exist.


Now, whether your use case exists within the common knowledge area or within the long tail


is really how you can hazard a guess at whether you need to fine tune a retriever model or not.


So if we just modify that chart a little bit, we get this.


So we have the same thing, common knowledge on the y-axis.


And what we have on the right, I've just renamed it.


So the ease of finding a model and or data.


OK, so based on how niche your use case is, the harder it is going to be to find data for that niche.


And the less data there is out there, the less likely someone else is already trained or pre-trained a model.


So most of the pre-trained models out there are trained on a very generic, broad range of concepts.


They're trained on Wikipedia pages or something like that.


So that's fine.


You know, if you compare cats versus dogs or even seed versus Java,


the model is probably being pre-trained on something similar to that and it might be able to figure out.


But if your use case is more specific, like you have, I don't know, like some very specific financial documents or technical documentation,


something along those lines where not many people understand the content of that documentation.


And there's not, you know, it's not general knowledge or it's not easily accessible on the Internet.


In that case, you will probably need to train or fine tune your own model.


So if that is the case, how do we do that?


Well, to train a retrieval model, we need pairs of texts.


So we need questions and relevant context.


So that's what you can see here. We have question A, context A.


They're both related and they both end up in the same sort of area.


That's what we need to teach our retrieval model to do.


We tell our retrieval model, OK, question A and context A.


Process them and output a vector for each one of those.


And then we want to look at those two vectors and say, OK, are they similar or not?


If they're not similar, we tell the model, look, you need to figure this out and make them more similar.


If they are similar, then great, good job.


That's what we're optimising on.


We're optimising and minimising that difference between similar pairs and maximising the difference between dissimilar pairs.


Now, our data is going to just contain all of these rows where we have the question and context pairs.


We don't need labels because we are going to be using multiple negatives of ranking loss, which we'll discuss in a minute.


There's also a video on that if you do want to go into a little more depth.


So to train our model, we're going to be using the Squad2 dataset from over here.


And in there we have our questions and we have context.


OK, that's what we're going to train on those two pairs.


So let's take a look at what that process looks like.


So as I said, we're going to be using the SquadV2 dataset.


We're going to be pulling that from Huggabase dataset, so you may need to pip install datasets if you do not have that already.


And this is how we load the dataset.


So we've got SquadV2 and we're getting the training split of that because there's also validation split that we will use later.


And from there, what we will see is we'll get an ID, title, context, the question.


So only the question and context is really that important for us.


So we can go down and just a few examples here of different samples or rows from the dataset.


And like I said before, we need to take the question and context pairs from the dataset.


So to do that, we're taking them here. We're also creating this input example or list of input example objects.


Now input example is just the data format that we use when we are training with the sentence transformers library, which you can see here.


Now, again, if you do need to install that, just pip install sentence transformers.


And TQDM is just the progress bar that you see down here.


So what we're doing is just appending a load of input examples where we have the question and context.


We don't have any label here because we are going to be training with multiple negative ranking loss where we don't need labels.


Now, also because we're using that type of loss, MNR loss, we need to make sure that each batch does not include duplicates.


Now, the reason for this is when we train with MNR loss, we're going to be putting everything in or training in batches.


And the model is going to be looking at two pairs like this. It's going to take the question and it's going to say, OK, for that question,


this context here needs to be the most similar and all of these contexts need to be as dissimilar as possible.


Now, the problem with if you have duplicates in your batches is that you have, let's say, the exact same, maybe not the exact same question,


but you have the exact same context down here. Now, your model is optimizing to make all of these as dissimilar as possible.


But also this one here, even though it's exactly the same as the one that optimizing to be more similar.


So you really need to try and avoid this. It's OK if it happens occasionally in the odd batch, but you need to avoid it as much as possible.


So that's why we use this no duplicates data loader. This will make sure we don't have any duplicates within each batch.


So we'll go down. We need to initialize a sentence transformer.


Again, this is the same as what we usually do. I've written BERT here, but we're actually using the Microsoft MPNet model.


Now, MPNet is really good for sentence transformers in general.


If I did this with a BERT model, I think the performance is two percentage points less than if I use an MPNet model.


It's not huge, but it does make a difference. So it's good to try both if you want.


But this one is the one I'd go with. So here we've initialized the model.


We also have this pooling layer. Now, the pooling layer is important.


That is what makes a sentence transformer rather than just a normal transformer.


And it works like this. So we have our sentence.


It will get tokenized and split into many tokens all down here and put into BERT, MPNet or some other transformer.


And on the output of that, we get all these token vectors.


Now, all these token vectors represent our sentence, but there are loads of them.


We want a single vector to represent a single sentence.


So we use this pooling layer and this pooling layer takes all of those vectors.


And takes the average in every single dimension. So we take the average and we get our sentence vector.


That's both how and why we use the pooling layer.


So we have that. We can come down and see our transform model. We have the pooling layer. It's good.


Like I said earlier, we are going to be using MNR loss for training.


And that is the batching thing where we want to get the pairs together and then we rank all the other contexts as dissimilar.


So we initialize it like this using the sentence transformers library and then we're ready to train our model.


So it's really not that complicated, particularly if you're using sentence transformers library. It's very easy.


So we warm up for 10% of the training steps. That's a pretty standard number for sentence transformers.


You can modify it a little bit, but 10% is usually pretty good. It just helps make sure we don't overfit.


And the same for EPUBs. Almost always set that to one when you're fine tuning sentence transformers.


Otherwise it really does tend to overfit. So it's usually a good idea.


It shouldn't take too long to train. It's 30 minutes for me here. It might take a bit longer, but really it's not training on too much here.


I think we have like a hundred thousand. I think we have a hundred and thirty thousand training samples in the spot data.


Now, another really important thing is evaluation.


So once we've trained our model, how do we know that it actually works? Is it performing well? We don't know.


So we need to measure how accurately our model is retrieving the correct context for a particular question,


which is slightly different to other operations or evaluation metrics that we use with other language models.


And to evaluate the retrieval performance, we use this information retrieval evaluator.


Now, from this we're going to be using the map at K metric, which is in short an average precision value


or fraction of returned context that are relevant to the question we asked.


And the at K component of that is just saying we are going to consider the top K return result.


So if it's at 10, as in K is equal to 10, we're going to return 10 context.


And then we're going to calculate that metric from those 10 return contexts.


Now, by default, this evaluator is going to be using that map at K metric.


So we initialize it like this. So information retrieval evaluator and it needs data.


We're going to be using validation set of the same data we used before. So the squad V2 data set.


And it looks the same. We have ID, title, context, question, answers.


We at the moment, we're going to need ID, context and question.


Now, this evaluator needs us to map relevant questions and contexts using those IDs.


So what we're first going to do is we're going to convert this into a panel of data frames.


I find it a little easier to work with what we're going to be doing here.


So we're converting it to a data frame here. We see we have context ID question, which is all we need.


Now, we need to assign a specific or a new ID to the context.


Because at the moment, if you look here, we have an ID and it's shared by the context and also the question.


And another thing is the ID for the context, like here, even though the context is the same, the ID is different.


So what we're going to do is use these IDs for the questions because all the IDs seem to be unique.


I'm going to create a new ID for each context. So we're going to deduplicate that data frame.


So we have question and ID, nothing else.


And then we're just going to append this con onto the end of our ID.


So now we have unique IDs for all of our contexts as well as our questions.


And what we can now do is merge or form an inner join with our no-dupes data frame and the original data frame.


So do that. And now we have a unique ID.


So this ID Y for each of our contexts. So you can see ID Y is not changing where we have these duplicate contexts.


And then we have ID X for questions. So this is what we need for our evaluator.


So we need to reformat this into three different dictionaries.


So we have IR queries, which is a mapping of question ID to question, like the text.


We have IR corpus, which is a mapping of the context ID to the actual context.


And we also have IR relevant docs, which is a mapping of the question ID to relevant context ID.


So you can have multiple context IDs for a particular question.


But in this case, it is actually just one to one mapping.


So we first create the IR queries dictionary.


So it's literally just the ID as a key and the question as a value.


So we create all of those key value pairs.


Same again for IR corpus, exactly the same, but this time with a context.


And then if we come down here, so this one is slightly different.


So this is mapping the question ID to a set of relevant context ID.


Now, we could map these directly. But what I've done here is if you use this same script and you have an example where maybe you have multiple contexts for each question,


this will handle those. And so you will get a list or a set of multiple context IDs in here rather than just one, if that is relevant.


And we see that we have multiple, in our case, we have multiple questions that map to a single context ID.


So it's actually many to one rather than one to one, like I said before.


Now, with each of those three dictionaries, we can visualize our evaluator, which is just passed in like this and we evaluate.


And that's really simple. There's nothing else to it.


And so the map at K performance for that is 0.74, which is good.


If you compare it to some of the set of the art retrieval models.


So this is also using MPNet and it's been trained on a lot more data, by the way, as well.


So it's more more general. This is getting 76 percent.


So a little bit better than ours, like two percentage points better. But this is the SQUAD dataset.


This other model here has, I'm pretty confident, has been trained also on the SQUAD dataset.


So it has already seen that data. If you do the same thing, but for your own more niche dataset,


I think most likely you will get the model that outperforms any other pre-trained models.


But of course, definitely just test it, evaluate what you get.


OK, so we have fine-tuned our retrieval model now.


But we can't really use it unless we saw the vectors that we create with our retrieval model somewhere.


So we need a vector database. Now, to do that, we need to take a few steps.


So we have already done a few. OK, so we have trained, specifically we trained our retrieval model so we can cross that off.


We've also downloaded a load of context. So the SQUAD context, we'll use a validation set for that.


Now, the next step on this side is to encode those contexts, which we will do in a moment.


And then over here, we also need to initialize our index for our vector database.


So we're going to do that and then we're going to take those encoded contexts and now initialize index.


And we're going to populate that index with our encoded context vectors. So let's get started with that.


OK, so in a new notebook now. So if you're following along and maybe your model is still training or fine-tuning, that's fine.


You can actually just download the model that we trained here. So it's just Pinecone, MPNet Retriever, SQUAD2.


That's the model that you saw a moment ago. If you want to try to pair it on as well, you can replace this with BERT.


And that's also there. So we have that model. We can see it looks same as what we had before.


And I'm also just going to reload the validation data set.


Now, I'm going to extract all of the unique contexts because we have, like we saw before, the SQUAD data has many copies of the same context.


And we don't want to have duplicates in our index of the same vectors because if we're searching and we compare the distance to our query vector and all these contexts,


and we have like five that are in the exact same position, we can return all of those.


So we don't want to do that. So we need to remove duplicates.


OK, so we're looping through the whole data set. We're checking if context is already within this unique context list that I've initialized.


If it isn't, we add it to that list. And we also add the ID. So remember, the question ID, context ID was shared.


We're just using that ID this time. We're not using the unique context IDs that we created before.


So we loop through and then obviously when it sees that context again, it's not going to add that different ID to the unique IDs list.


And then we can filter using the Hug & Face data sets library here to only keep the IDs or the rows which have an ID from the unique ID list.


So with that, we end up with just one of each context.


You can see here we only get twelve hundred rows out of that, which is much less than the full squad validation set.


Now, what we want to do is encode those contexts and the context vectors.


So we use our model that we've initialized and we use the encode method.


Now, we also convert that vector. So this, I think, outputs a PyTorch tensor.


We convert into a list because we are going to be sending all this to our Pinecone Vector Database index through an API.


And for that, we want to have them in a list format. So we encode the context and now we can see, OK, we have all the other features that we had before.


Now we also have the encoding feature. So we move on to the vector database.


Now, how do we use this vector database? We first need to install the Pinecone client here.


So again, we're using the Pinecone Vector Database. So we come down and we can import Pinecone.


So this will just import the way that we interact with our Pinecone Vector Database and we then initialize a connection.


So for this, you do need a free API key. So it's free. You don't need to pay anything for that.


So you just go to app.pinecone.io and you create an account.


And then with the API key that you're given, you enter in here.


So just in there, Pinecone init API key. And you also set your cloud environment. So US West 1 GCP.


And then what we can do once we've initialized that connection is we create a new index.


So to create that index, we only need this little chunk code. We need this dot create index method.


And we specify the name of the index. I'm going to just call it squad index. You call it whatever you want.


And the dimensionality of our vectors as well. So you can see here it's 768.


And you can also define the metric. Now you could also use Euclidean distance, but we are going to stick with the default, which is cosine.


And yeah, that's all you need to you only need to run this. But at the same time, because I have multiple indexes and I'm re-running this code and testing it.


I added this in. So I'm just saying if squad index is not already existing.


So I'm checking that it is not already running that and then creating it.


And then we connect. So specifically to our index run, just our pinecone database as a whole.


We connect. So we use pinecone index and specify the index that we'd like to connect to.


And then from now on, we just use this index object for everything we're doing.


So here we are just preparing our data to insert into pinecone, which is just uploading to our pinecone index.


So when we are uploading data into pinecone, we have three components.


We don't need all of them, but I'll go through them. So we have a list.


And within that list, we have these two boards. In there we have the ID of the vector or the entry record, whatever.


And then you have your encoding or the vector itself. And then optionally, you don't need to include this so you can remove it if you want.


You have a dictionary of metadata. So this is just key value pairs that you can use for things like metadata filtering.


Or if you want to return a particular piece of data with your search, you can include that in there as well, which is why I'm using it for here.


You don't need to do this. You could store the text locally. But I'm just doing it like this because it's a little bit easier for this little script.


And with that, we just upsert. So we just run this index upsert. I'm doing it in batches.


You can probably increase the batch size here, but I'm just sticking with that. And it is pretty quick anyway.


So 30 seconds is no problem. So with that, our index, so our vector database and our retriever are ready.


So we can actually begin querying and asking a question and returning a relevant context.


OK, so we're on to another notebook for this. So again, I'm going to initialize the retriever model and I'm going to input pangcone,


initialize the connection to the index, to the squad index again. Now, this is pretty straightforward, fortunately.


Now we're at the end. All we do is we take a question. So I'm going to say when were the Normans in Normandy?


And we just write this. So model and code include our query within a list here, because, for example,


if you had multiple queries, you might have that in a list, something else up here.


And in that case, you would not have to add these square brackets. But in this case, we just have a single query.


And we create a PyTorch tensor, which is our query vector, and we convert it into a list because we're going to be sending it through the pangcone API again.


And what I'm doing here is I'm querying. So this is going to the pangcone database. I'm querying.


I'm saying I want to find the most similar vectors or context vectors to this query vector or question vector.


I want to return just the top two contexts that you find. And I want to include a metadata in that response because in the metadata I included the text.


So we can come down here and we see, OK, this first example isn't actually what we want.


So in here, what we asked when were the Normans in Normandy? This doesn't actually answer that question.


But then the second one, so if I open this up in the text editor, come down here, we have the second question.


So the Normans were the people who in the 10th and 11th centuries gave their names Normandy.


So we can assume that they were probably in Normandy in that time. So that's when they were there.


So we get the correct context in second position for this one.


We also get the scores. It's a good score. It's high.


And then we have another question here. So we've got three questions. We'll go through each one of those pretty quickly.


How many outputs are expected to reach input in a function problem? And we do actually return the correct answer for that one straight away.


So a function problem is a computational problem where a single output is expected for every input.


So that's our specific answer there. OK, we get a really high score for that.


So it's pretty confident that this is correct. And then a lower score for the other ones where the actual question is not answered.


So one final one here. So I changed the wording a little bit for this one because I don't want to be just doing a keyword search.


So I put who use Islamic Lombard etc. construction techniques in the Mediterranean.


And I modified that a little bit and we do actually return the correct answer straight away.


Confidence 0.6. I actually compared it to the other ones which are much lower.


That's pretty good. That's a good separation there which is what we want to be looking for.


So that is a... I know it's pretty long. There's a lot of components and moving parts there.


But that's the open domain question and answering pipeline or at least the vector database retriever components of that.


We've also had a look at how we can fine tune our own retrievers. So we covered quite a lot.


So I mean with all of that you're ready to just go ahead and implement what I think are probably two most crucial components in open domain question answering.


If you don't have a good vector database and good retriever model you are going to be returning not good context to your reader model.


And if you don't have good context for your reader model your reader model isn't going to give you anything good.


So these two are probably the most important parts.


If your reader model is rubbish maybe it gives you a kind of weird span or answer but at least you've got a good context to go off of.


So your users are at least getting some relevant information.


Now I think one of the coolest things about open domain question answering is just how widely applicable it is.


Basically any company across so many industries across the world can use this if they have unstructured data that they need to essentially open the doors to


for their staff or their users.


If you want to get data or information to someone which is a big part of most jobs.


If you want to get data information to someone more effectively and using a more natural form of search and question answering this is probably applicable to you and maybe your company.


Fine-tune Sentence Transformers the OG Way (with NLI Softmax loss)

Hi, welcome to the video.


We're going to be covering how we can train a S-ber model


or a sentence transformer or sentence-ber model


using the...


what is kind of like the original way of training these models


or fine-tuning these models, which is using Softmax Loss.


So let's start with just a quick overview of the training approach.


Now, using the Softmax training approach


is part of what we could call the Natural Language Inference


approach to fine-tuning these models.


And within that sort of category of training, we have two approaches.


We have Softmax Loss or Softmax Classification Loss,


which we're all going to cover.


And then we also have something called a Multiple Negatives Ranking Loss.


Now, in reality, you probably wouldn't use Softmax Loss


because it's just nowhere near as good as using the other form of loss,


the Multiple Negatives Ranking.


I'm going to call it MNR from now on.


So MNR is more effective, but Softmax Loss is sort of the original


and that's why we're covering it here.


So we're just going to go through it.


I'm not going to go into too much depth.


I'm going to just kind of go through it very quickly.


So when we're training these models, we can either use what is called


a Siamese network or a triplet network.


Now, what you can see right now is that it's a Siamese network.


So we have almost like two copies of the same BERT,


so they're like Siamese twins.


And the idea is we would have two sentences, sentence A and sentence B,


and we would feed both of those through our BERT model


and produce the token embeddings that we usually get from BERT.


And then we use a pooling operation, so like mean average pooling.


And then from that, we get our sentence embedding in U and V.


And what we would be doing is optimizing to try and get those


sentence embeddings as close as possible for similar sentences.


And then for dissimilar sentences, run them to be as far away


from each other as possible.


So that's kind of like the start of the model, but it's not the full model.


We continue.


And what we're going to do is concatenate those two together.


So U plus V here.


And then we're also going to do this other operation here.


So we're going to take U and V, and we're going to get the difference between them.


So this is just a positive number here.


We're taking the magnitude here, these bars.


So we're just getting a positive number, which is a difference between the two vectors.


And we also concatenate that.


So we create this big vector, which is U, V, and then we have U minus V at the end.


And we take that vector, so that's what you can see over here,


and we feed it into a very simple feed-forward neural network.


So the feed-forward neural network, each one of these sentence embeddings,


we're going to have the dimensionality of 768.


So obviously the dimensionality or input dimension of our feed-forward neural network is 768,


which is powered by 3.


And then the output are our three output activations here.


Now, if you watched the last video or you read the last article,


you will remember that we had three labels in our training data.


So in our NLI training data, we had its entailment, neutral, and contradiction.


So those sentence pairs that we're trying to classify, are they relevant to each other or not?


And then we have the true label over here.


And then we just optimize using cross entropy loss, which is what you can see over here.


The reason why this is called softmax loss is because there is a softmax function within the cross entropy loss function.


So that's the processor at high level.


So let's jump in.


We're mainly going to focus on how we format our data to put into here.


And then we're going to move on to how we actually train all of this using the sentence transformers library.


That's going to be our main focus.


But we'll just very quickly run through the code in PyTorch.


So you can just see how it works.


And if you really want to dive into it, you can obviously just take a look at the code and figure out.


It's not hard to read.


So let's jump into it.


OK, so the first thing that we want to do, I've added a little note up here.


This is just an information of what is in our data or the labels of our data.


We're going to have a look at our data anyway now.


So I'm going to use the Hugging Face data sets library.


So we're going to import data sets and we're actually using two different data sets using the Stanford Natural Language Inference or SNLI data set.


And also the multi-genre NLI data set as well.


So SNLI, we just put that in.


So it's data sets load data set.


And then it's called SNLI and we want the training subset of that.


So train, sorry, not subset, split.


And then we can have a look. What do we have inside the data?


OK, so we have these three features.


So these are columns or you can call them columns if you want.


So we have the premise, hypothesis and label.


Now in those previous diagrams we saw, we saw sentence A, which is the premise, sentence B, which is the hypothesis.


And then we saw labels at the end. So it's just the same.


Now, if you want to have a quick look at one of those, we just have this.


So we just get, what you can see here.


So what label do we have here?


We have label one, so we can compare. That's neutral.


So the premise and the hypothesis could both be true, but they are not necessarily related.


And then here we see the person is jumping over a broken down airplane.


The person is training his horse for the competition.


So they could both be about the same topic, but it's not necessarily about the same.


They don't infer each other.


So if we may try and find one that is a contradiction or something else.


Why did I respond again?


OK, so this one is a contradiction.


So a person on a horse jumps over a broken down airplane.


A person is a dining order anomalous.


So those two things are on about the same topic. So they're a contradiction.


And then the other one we have just, I think if I do, we should find one.


We have this one.


A person is outdoors on a horse.


So this here would infer, sorry, this here, this premise infers this hypothesis.


So that's the data.


And like I said, we have two of those data sets.


We have SMI and MNI.


So MNI, we load it the same way.


So data sets, load data sets.


It's from the glue data set.


And then the subset is MNI.


And again, we want to split to be equal to train.


OK, and if we just have a look, we'll see very similar format, but not exactly the same.


Let's see, we have my hypothesis label, then we also have this index.


We need to merge these two data sets.


We need to reformat our MNI data set a little bit.


So first thing we do, we write MNI and we want to remove that column.


So MNI remove columns and we are doing IDX.


OK, and let's make sure it works.


And we see now we don't have that.


And if we try and merge, you still get an error, which is annoying, but it's fine.


So I'm going to call it data set because data sets adopt concatenate data sets.


And we just pass them both to the list.


So SMI, MNI, and we're going to get this error.


OK, so the schema, so the format of the data set is different.


Even though they both contain the same columns, I think one of them has a slightly different format.


Like one of them allows you to have nulls that it does say here.


Right, so they both have slightly different formats.


So to fix that, we just want to change the schema of one of those data sets.


And all we do for that is we then change the SMI data set.


And say SMI cast features.


Just cast maybe.


And MLI.features here.


Right.


Yeah, and then we can actually do this now.


OK, so now we have our data set.


We can look and see, OK, we now have 942, well 943, basically 1000 rows there.


Now inside this data, we actually have some rows that we don't want.


So we should have labeled 0, 1, and 2.


Which we have up at the top here, 0, 1, and 2.


But there's actually some rows that have the label minus 1.


And all these are just erroneous rows.


We don't actually want those in there.


It's where someone couldn't figure out what to actually rate that sentence pair.


So what we're going to do is just remove those.


So we write data set equals data set.


And we're going to use filter.


And then we just write a lambda function.


This lambda function is going to select rows where the label value is not minus 1.


So we're going to say false if the label value, so label,


is equal to minus 1.


So we're going to filter those out.


And then obviously we want to put else true to keep the other columns.


OK, so let's just print it out and we can see.


So we have 942.8 there.


And here we have 942.0.


So removed, I think it's like 700 or so rows.


So if we are using the sentence transformers way of training the models,


this is pretty much all we have to do.


There's one more step that we have to take,


which is to convert the data into input examples or a list of input examples,


which we'll move on to in a moment.


Now I'm going to quickly just cover the other training approach using PyTorch.


But I mean, it's quite complicated.


And at least when I was trained using that approach,


the model was nowhere near as good as when I trained it using sentence transformers.


So I wouldn't recommend it.


But if you're interested, this is how we do it.


So let me switch over to that notebook.


So if I come over here.


And okay, we're going to see it's basically doing the same thing.


We're loading data sets and come down.


Now, it's a difference.


So the difference here, so we're importing mainly the better tokenizers


while focusing on here.


We come down and then here.


So we're filtering, which is what we did before.


Nothing new there.


But here we're doing something different.


So here we're actually tokenizing our text.


Right. So we're using this map.


We're using this map function here.


We're tokenizing both the premise sentences and also the hypothesis sentences.


And we get the input IDs and attention mask out of those.


And if we have a look down here,


we'll see that this is what we end up with at the end there.


So I've removed all the other features and all we have are the labels.


And then we also have the input IDs and attention masks both our premise or sentence A


and also our hypothesis or sentence B.


And then after that, we need to do this as well.


This data set dot set format.


And we use, because we're using PyTorch, we set it into a torch format.


Okay.


From there, we, you know, typical PyTorch stuff here.


So we're setting up a data loader using batch size 16.


That's what they use in the expert paper.


And then if we come down.


This is all just examples.


So I'm actually going to go a little further down.


So to.


To here. Okay.


So here I'm defining the, you remember before in that graph,


we had the, we had the, we passed sentence A, sentence B.


They went, both went into the BERT or the Siamese BERT.


And then there was this pooling method, which took our token embeddings,


which are 512 768 dimensional vectors and compress them into just a single 768 dimensional vector.


That's what this function here does.


When we're using sentence transformers, we don't need to worry about this.


Sentence transformers, the library, by the way, the framework.


That's probably a bit confusing.


But I mean, when I say sentence transformers or using sentence transformers,


I mean the framework or library, which we're going to cover soon.


But obviously they're also the models.


That's the name of the models.


So here I'm taking the, the mean pooling.


It's taking the average of those values and excluding values that are padded,


which is why we're not just taking the average straight.


We are removing those attention mass values.


We go down, we move our device.


So we check if we have a CUDA enabled GPU and move our model to it if we can.


And then here, these are the layers we use.


So I told you before, we had that where we concatenate our U and V vectors,


the sentence embeddings, and then we pass them to a feedforward neural network.


And that feedforward neural network is the size of our sentence embeddings


multiplied by three, and it outputs three labels or classes.


And then we also use a cross-entropy loss function


between what the feedforward neural network outputs and our actual labels.


So after that, you know, this is what I mean.


There's quite a bit of code when you go to sort of like the very manual


PyTorch approach rather than using the sentence transformers library.


So here we're getting this get linear schedule with warmup.


So that's just saying for the first 10% of our steps,


we're going to warm up the learning rate.


So we're not going to go full on training at 1e to the minus five.


We're going to warm it up a little bit.


Now, in the spec paper, they use 2e to the minus five.


For me, it just kind of bounced around a lot, so I halved it.


But, you know, this is just, I mean, if you can get working with 2e to the minus five,


that's what they use in the paper. So it's probably better.


And then they only train for one epoch as well.


And then also here I'm using the add and move weight decay.


And then this is the training loop.


So TQDM is just a progress bar. We do one epoch.


We make sure our model is on training mode.


We initialize a loop, which is going to get all the batches from our data loader.


And then we're just getting all the data out. This is just PyTorch stuff.


Getting our U and V sentence embeddings.


Then here we're getting the, actually, sorry, so U and V here, actually, token embeddings.


Here we're converting them into sentence embeddings.


And then we're getting the U, the absolute value or the difference vector.


Here, concatenated is all together. So we're creating that concatenated vector


that we then feed into the v4 neural network.


And then we optimize based on the loss here.


And that's pretty much it. And then we're saving the model down here.


Okay, so, yeah, that's how we train it in PyTorch.


You can see here I was messing around, seeing if I could see what happened if I did two epochs.


It's better to just stick with one.


Even though the loss was lower, in the end, the performance wasn't any different.


So I would train for one epoch.


Okay, so let's go back to the code and we'll work through the actual training with sentence transformers,


which is what I would recommend doing.


Okay, so I said before we had the list of input examples.


So input example is just a data format that sentence transformers library uses.


So we just want to write from sentence transformers, import input example.


And then all I'm going to do here is write from dqdm.auto, also import dqdm.


So this is just for our progress bar. So we can see what is happening.


And then in here, we just want to actually want to create our training examples first or training samples,


whichever you want to, whatever you want to call it.


It's going to be empty list.


And then we literally just follow through all of our training data, through our data set and extracting what we need from it,


which is just sentence A, sentence B, and the label.


So right, for row in, I'm going to put tqdm train samples.


So just adding tqdm there so we have a progress bar so we can see where we are.


All we need to do is write train samples that are penned, input example.


And before you get confused, this should be a data set, not train samples.


So data set is what we're looking through our data set, right? Not the empty list.


And then inside our input example, we have two variables of text and labels.


So you have to pass your text, which is the input text that you're going to process into your model.


So we go row, promise, and also row, hypothesis.


So they're just our two features from the two text features from our data set.


And then here we also want to label. So label is just row, label.


Just a feature names from our data set, which you can still see up here.


Now we process that. It can take a little bit of time, so I'm going to take too long, fortunately.


And then from there, we need to, you remember before, well, one row quickly went through.


We had the PyTorch data loader. We also need a data loader here as well.


Sometimes you can use special data loaders from the sentence transformers library, which are quite good.


But for this, we're just using a normal PyTorch data loader. So we need to import torch for that.


We can just write from torch utils data import data loader.


Same as the paper we're using batch size of 16 and the data loader is just data loader.


We pass in those train samples, specify our batch size.


And if you'd like to shuffle, which in this case we will, you also put shuffle. So shuffle equals true.


And that should work. OK, so now we have a data loader.


And what we do now is initialize our model using sentence transformers.


Sentence transformers uses modules to set up the model.


So we're going to have a transformer module, which obviously you got to have BERT.


And then we're also going to have a pooling module, which is for our mean pooling layer.


So from sentence transformers again, we're going to import the models.


And what is this one? Sentence transformer.


Yeah, we initialize those two modules. So we have a BERT module.


So models transformer. And then here it's just it's using the the Huginface models.


So we can put anything from Huginface on here.


I'm going to use BERT based on case. And then we also want our pooler.


So our pooler, models again, pooler, pooling.


And then we have BERT and we want to get the word embedding dimension.


So get word embedding dimension, which is the 768 of our of our token embeddings.


And then, of course, about sentence embedding as well.


And then we also want to set the type of pooling that we are going to do.


So pooling mode. You see that we have these different ways of pooling.


So you have CLS token maximum. This one I've never actually seen use square root.


The length is interesting. And then we also have this one.


This is the mean pooling and we're going to use that. OK.


So that is. There are two modules and then we just want to initialize our model.


So we write sentence transformer. And what you can you can do, by the way, is this is how you would actually say you have a sentence transform model that you want to load.


You would write the sentence transform name in here.


So like all MP Nets, whatever it's called, as you do that as well.


But you can also load or initialize model using the modules that we just initialize.


So write BERT followed by the pooler. And then key details of that model in there.


OK, so ignore that. So this bit here. So these are this is our sentence transformer structure.


You can think of it as a structure. So our transformer, we're using BERT model, the maximum sequence length.


And then in here we have our pooler and we have the word embedding dimension that we'll expect.


768. And then you see here we have those different pooling modes and we are using pooling mode mean tokens, which is true.


The rest are false. And then from there, so we we also need to initialize our loss function, which is pretty straightforward as well.


So again, from sentence transformers, I want to import losses and there are plenty of different losses that you can use.


You can just look on that documentation. But we're using softmax loss.


So what we want to do is write loss equals losses and we write softmax loss.


And then in here, so you think, OK, our loss function, what does it need? So we pass in the model so it can get the model parameters from that.


So first we do model equals model. And it also needs the embedding dimension.


So 768 again. So it needs on this sentence embedding dimension.


And there we just want, what is it? So the model gets embedding dimension again, gets sentence embedding dimension.


OK, and then we also need to pass, OK, how many labels are we going to have in our model?


We already know it's the number of labels. We already know it's three.


I'm sure you can get that dynamically from the data set if you want as well. But I'm set up for three.


And I think that's it. So we have our loss model and our data.


So I think that we should be OK to start training.


So I'm going to say we go for, OK, one epoch. We want to say how many warm ups steps do we want.


So again, the 10% warming up that we use.


So we just want 0.1 multiplied by the length of our data set.


So length of the data set. Yeah, and I'm just going to, we need that to be an integer value.


So I'm just, you know, it's quite rough, rounding very roughly there, but that's fine.


And then we want to just start training our model. So right, model.fit, it's like TensorFlow.


And we use train objectives. So in here, we need to pass a list which contains a single tuple, which is our loader.


So the data loader and the loss. So I think with this, you can, if you have multiple train objects,


but like another loader, another loss and keep going through there. That's why we have a tuple within a list.


Then you have your epochs, the number of warm ups steps, which is just warm ups steps again.


So warm ups steps. We also need the, what do we need? Output path.


So where we're going to save the model. So I just put like, experts.


What do I call it? I think test B is what I've called it later on.


And oh, last one is show progress bar. Now, this is automatically true.


But when I zoom that, it just printed out loads of lines that it's printing to a new line every single update.


So I just set to false. So it wouldn't do that because it's obviously quite annoying.


So, yeah, that's how you train model. I'm not going to do it again. I've already done it.


I've already trained this expert test B model.


So what I'm going to do is switch over to that notebook where I trained it and show you those results.


OK, so this is a notebook. Pretty much what we've just covered. We're going through that again.


And then here, so we have the training, the training time as well.


Something I didn't mention just now. It's one hour and 15 minutes for me on RTX 3090.


You know, reasonably fast. It depends on what you're training on.


Yeah, so it's quick. So I define these sentences just below random sentences.


Like complete nonsense. But some of them do align. So see this one.


One of the things she saw her raw fish and rice change position.


And this one seeing her sushi move, weaving with spaghetti.


And where is the other? Knit with noodles and dental specialist with construction materials.


And same again, dentist with chewing bricks.


So there's some that are kind of similar, but they don't share any of the same descriptive words.


But they are, they kind of mean the same thing roundabouts.


So with our model, so we have loaded the model here.


We can just do, so if you if you've saved the model, which it does automatically here, you just take this.


You take that, you come down here and you write sentence transformer.


And then in here you do that, right? And then you would put that in the model variable.


Right, so that's all I've done there. So move that.


So loading model, I'm going to use it to encode those sentences, which is just in the list and create embeddings.


So these sentence embeddings. And then from there, what I'm doing, getting the cosine similarity again, using sentence transformers for that.


This makes it easier. And I'm just comparing all those sentences.


So I'm just, you know, a very qualitative view on how these embeddings are doing.


And you see that we get these results here.


So it's not, it's pretty good actually. Like it is getting the right ones.


So this seven and five, nine and one. And I think four and three are the ones that we wanted to get.


And they are, in fact, the highest three rated scores.


But a lot of these other non pairs are still rated kind of high.


And like I said before, right, softmax loss is not the best way of training your model anymore.


Or fine tuning your model anymore. There's other ways like MNR loss.


So let me show you some of the charts from MNR loss and you'll see the difference.


OK, so we have this one is just for BERT. You see they're all very flat.


It does actually get almost, it gets two of the correct answers within its top three, but not all of them.


But it's very flat. Like all the values are very near the same value, which means it's hard to differentiate between similar and not similar, which is not what we want.


But obviously BERT hasn't been trained for this, so you can't expect it.


And this is my PyTorch model. It's getting better than the BERT model.


But the performance still isn't there compared to the sentence transformers trained model.


Which is not here. This is the actual, the one that they trained, sentence transformers themselves.


You can see there's better differentiation between general values here.


But then if we compare that to, so this is an MNR model that I have trained using the same sentence transformers. We use sentence transformers here.


We come down, we see a big difference. So these are our similar pairs and they stick out so much more than they did with the other models.


Everything else is scored very lowly. But these stick out a lot.


And that's really the difference between models.


This is much better because it separates those similar and dissimilar pairs very well. It's just a lot more accurate.


So that is, I mean, that's my MNR model as well. The actual sentence transformers MNR models are much better than this.


But yeah, I think that's pretty much it. That's it for this video. So in the next one we are going to, as you probably have guessed,


we're going to have a look at how we can use MNR loss or multiple negative ranking loss to build a model.


Which I think personally is a lot more interesting.


Softmax loss is pretty interesting, but it's not particularly intuitive. And even the sentence transformers authors, the expert authors said the same thing.


That's actually where I got it from. It isn't very intuitive when you think about it. It's kind of hard to understand why it works.


Because we have that weird concatenation at the end and we're classifying and it seems strange. MNR loss is much more intuitive and it makes a lot more sense.


And I think it's more interesting and you get way better results with it. So we're going to cover that in the next video.


So I think that should be pretty interesting. But for now, that's it on softmax loss. Thank you very much for watching.


Zero-Shot Image Classification with OpenAI's CLIP

Today we're going to talk about how to use CLIP for zero shot image classification.


That is image classification without needing to fine-tune your model on a particular image dataset.


For me this is one of the most interesting use cases that really demonstrates the power of these


multimodal models. Those are models that understand different domains of information.


Like for CLIP for example that is image and text. But how about other state-of-the-art


computer vision models? How do they perform in comparison? Well these other computer vision


models they are characterized by the fact that they really focus on one thing in an image.


So for example let's say we had a computer vision model that was trained to classify an image as to


whether it contained a dog, a car or a bird. Okay that computer vision model doesn't really need to


know anything other than what a car looks like, what a dog looks like and what a bird looks like.


It can ignore everything else in those images. And that's one of the limitations of these


state-of-the-art computer vision models. They have been pre-trained on a huge amount of data but


they've been pre-trained for classification of these particular elements. And even if you have


like a thousand or ten thousand classes in there that hardly represents the real world which has


a huge variety of concepts and objects for us as humans to understand and see and categorize.


So what we find with these computer vision models is that they perform very well on specific


data sets but they're not so good at handling new classes for example. For that model to understand


a new class of objects and images it will need to be fine-tuned further and it's going to need a


lot of data and it's just it's not very easy to do. Ideally we want a computer vision model to


just understand kind of not everything would be perfect obviously we're not quite there yet but


if it can just understand the world and the visual world relatively well then we're sort of on the


way to a more robust model. So for example this image of a dog we want the model to understand


that dog is in the image and a you know usually use convolutional neural network that sort of


model for image classification will know that there is a dog in the image but it doesn't it


doesn't care about everything else in the image. Ideally we want it to understand that there are


trees in the background the dog is running towards the camera it's on aggressive field it's sunrise


or sunset and that there are some blurry trees in the background the blue street. Unfortunately


classification training we don't get that. Instead the models are essentially just learning to push


their internal representations of images with dogs in all towards the same sort of vector space. That's


essentially how we can we can think about this and then for example earlier I said cars and birds as


well we could imagine so they have like a cluster of dog images they have a cluster of bird images


and they have a cluster of car images but they don't really have anything else in between that.


So this is ideal if we just want a yes or no answer for a specific data set and we want good


performance we can do that as long as we have the training data and the compute and the time to


actually do that. However if we don't have all of that and we just want good performance maybe not


state of the art but good performance across a whole range of data sets that's where we use Clip.


Clip has proved itself as an incredibly flexible model that can work in both text and images and


is amazing at what something we call zero shot classification. Zero shot basically saying you


need zero training examples for this model to adapt to a new domain. So before we dive into Clip


let's just explain this zero shot thing in a little more detail. So zero shot comes from


something called nshot learning. Nshot you may have guessed is basically the number of training


examples that you need for your model to perform on a particular new domain or new data set. Many


state of the art image classification models they tend to be pre-trained on ImageNet and then they


fine-tune for a specific task so that they have the pre-training and the pre-training basically


sets up the internal model weights of that model to understand the visual world at least within the


scope of the ImageNet classification set which is fairly big but it's obviously not as big as the


actual world. And then those models are usually fine-tuned on a particular data set and to fine-tune


that pre-trained image classification model on a new domain you are going to need a lot of examples.


Let's say as a rule of thumb maybe you need 10,000 images for each class or each label within your


data set. That may be excessive, it may be too little, I'm not sure but you do need something


within that ballpark in order to get good performance. We could refer to these models,


so these are like ResNet and BERT as many shot learners. They need many many training examples


in order to learn a new domain. Ideally we want to maximize model performance whilst minimizing


the n in nshot. Okay so minimizing the number of training examples needed for the model to perform


well. Now so as you know that CLIP is not achieving state-of-the-art performance on any


particular data sets or benchmarks. Other than one surprisingly without seeing any training data for


this particular data set, CLIP did actually get state-of-the-art performance on that one data set


which is surprising without seeing any of the training data but here we go this is how useful


this sort of thing is. Let's talk about how CLIP makes zero-shot learning so effective. So CLIP


stands for Contrastive Language Image Pre-training. It was released by OpenAI in 2021 and since then


it has done pretty well. We can find it in a lot of different use cases this is just one of them.


So CLIP itself actually consists of two models. I've discussed this in a previous video and


article in a lot more detail so if you're interested go and have a look at that. For now


we're going to keep things pretty light on how CLIP works but in this version of CLIP


those two models are going to consist of a typical text transformer model. So we're going to


have a text transformer model for dealing with the text encoding and a vision transformer model for


dealing with the image encoding. Both of these models within CLIP are optimized during training


in order to encode similar text and image pairs into the same vector space whilst also separating


dissimilar text and image pairs so they are further away in vector space. So essentially


in that vector space similar items are together whether they are images or text. Now CLIP


distinguishes itself from typical image classification models for a few reasons.


First it isn't trained for image classification and it was also trained on a very big data set


of 400 million image to text pairs with this contrastive learning approach. So from this we get


a few benefits. First for actually pre-training the model, training the model, CLIP only requires


image to text pairs which in today's age of social media they're pretty easy to get. Any post on


Instagram for example there's an image and there's usually a little caption of someone describing


what is in the image. We have sock photo websites, social media, you know just everything everywhere.


We have images and text usually tied together. So there's a lot of data for us to pull there.


Because of the large data set sizes that we can use with CLIP,


CLIP is able to get a really good general understanding of the concepts between language


and images and just a general understanding of the world through these two modalities. And as well


within these pairs the text descriptions often describe the image not just one part of the image


like okay there's a dog in the image but something else like dog is running in a grassy field.


Okay they describe something more and sometimes even describe very abstract things like the sort


of feeling or mood of a photo. So you get a lot more information from these image to text pairs


than you do with a typical classification data set. And it's these three benefits of CLIP that


have led to its pretty outstanding zero-shot performance across a huge number of data sets.


Now the authors of CLIP in the original CLIP paper they draw a really good example using


CLIP and the ResNet 101 model trained for ImageNet classification. Now CLIP was not


trained specifically for ImageNet classification but they showed that zero-shot performance with


CLIP versus this SETIVR model trained for ImageNet was comparable on the actual ImageNet data set.


And then when we compare them on other data sets that are derived from ImageNet so you have ImageNet


v2, ImageNet R, AlgeNet, ImageNet Sketch and ImageNet A. CLIP outperforms the model that was


specifically trained for ImageNet on every single one of those data sets which is really impressive.


Okay let's talk about how CLIP is actually doing zero-shot classification and how we can use it


for that as well. So CLIP, the two models within CLIP, they both output a 512 dimensional vector.


Now the text encoder, it can consume any piece of text right and then it will output a vector


representation of that text within sort of CLIP vector space. Then if you compare that text to an


image also encoded with CLIP what you should find is that text and images that are more similar


are closer together. So now imagine we do that but instead of, so we have our images from an


image classification data set and then for the text we actually feed in the class labels for


that classification task. Then you process all that and then calculate similarity between the