How to obtain [CLS] embeddings from fine-tuned BERT model (using Transformers Trainer)
I am sorry if this question has an obvious answer, I am a beginner and struggling to understand the inner workings of Transformers TrainerI have seen many answers explaining how to obtain the [CLS] embeddings from a BERT model when writing your own training loop using Pytorch. In that case, the embeddings are the last hidden-state/first element of the output tuple, which one can obtained by doing the following:model = BertModel.from_pretrained('bert-base-uncased')
input_ids = torch.tensor(tokenizer.encode("example sentence")).unsqueeze(0)
outputs = model(input_ids)
last_hidden_states = outputs[0]I am having some trouble understanding how I can obtain these embeddings when fine-tuning a pretrained model with the Huggingface Transformers Trainer object. When using that, I create a trainer object as follows:trainer = Trainer(

    model=model,

    args=training_args,

    train_dataset=small_train_dataset,

    eval_dataset=small_eval_dataset,

    compute_metrics=compute_metrics,

)And finetune the model by callingtrainer.train()Where can I find the [CLS] embeddings here?Thank you for your suggestions
2022-06-27T15:39:49Z
I think it’s the same? You pass the model object into the trainer and I believe it finetunes that model object as part of the train call. So that same model object should give you what you’re looking forI think the below code from your post above should work.input_ids = torch.tensor(tokenizer.encode("example sentence")).unsqueeze(0)
outputs = model(input_ids)
last_hidden_states = outputs[0]```