Last hidden state vs pooler output in CLIPVisionModel
Hi, I have an image tensor of shape (1, 3, 224, 224) and I want to get its embedding from theCLIPVisionModelwith theopenai/clip-vit-base-patch32weights.Now when I pass the tensor to the vision model, I get two outputs with the following shapes:last_hidden_state: [1, 50, 768]pooler_output: [1, 768]In one discussion thread, I read that choosinglast_hidden_state[:, 0]represents the overall representation of the image while in another thread I read choosingpooler_outputwill be better. I’m bit confused about this, can anyone please help me with this?
2022-11-18T08:12:38Z
Hi,Yes there are typically 2 ways to get a “pooled” representation of an entire image. One is taking thelast_hidden_stateand average them across the sequence dimension. So you could dolast_hidden_state.mean(dim=1)and use this as your image representation.An alternative representation is indeed thepooler_output, which takes the embedding of the first special CLS token from thelast_hidden_state, and applies a layernorm to it as seenhere.I’d say that average pooling all last hidden states vs. taking the CLS token’s representation give similar results. However the original ViT authors actually released anew paperin which they replaced the use of CLS token by average pooling and they got better results.