How to get the logits for the T5 model when using the `generate` method for inference?
I’m currently using HuggingFace’s T5 implementation for text generation purposes. More specifically, I’m using theT5ForConditionalGenerationto solve a text classification problem as generation.The model’s performance is overall very satisfactory after training, but what I am wondering is how I can get the logits for generation?I’m currently performing inference as is suggested in the documentation viamodel.generate(**tokenizer_outputs), but this simply outputs the IDs themselves without anything else.The reason why I want the logits is because I want to measure the model’s confidence of generation. I’m not 100% certain if my approach is correct, but I’m thinking that if I can get the logit values of each generated token and average them, I could get the overall confidence score of the generated sequence.Would anybody know how I could do this? Thanks.
2022-09-20T04:34:06Z
You can use the argumentsoutput_scores=True, return_dict_in_generate=Trueingenerate
2022-09-20T04:54:37Z
Hmm neither option seems to work. Isoutputs = model.generate(tokens['input_ids'], output_scores=True)the correct syntax?Apparently you have to set both arguments toTrue. I was only setting one. My mistake misreading the answer.
2023-04-18T05:09:15Z
you can use compute_transition_scores to get scores of each tokenfind more info hereGeneration