Scores in generate()
Hi,I was wondering why the length of theoutput_scoresis always +1 longer than themax_lengthin the output ofgenerate()? This seems to be not consistent with the documentationhttps://huggingface.co/transformers/internal/generation_utils.html#transformers.generation_utils.BeamSampleEncoderDecoderOutputI found thescoresfrom the output of thegenerate()function when settingoutput_scoresto beTrueis(max_length+1,)-shaped tensors or shorter due to the earlyeos_token_idwith each element of shape(batch_size*num_beams, config.vocab_size). The shape of theoutput.sequenceis(batch_size, max_length).
2021-02-04T10:22:00Z
Hey@Kylie,good observation! Sooutput_scoresshould max_length - 1. The reason is that the first token, thedecoder_start_token_idis not generated, meaning that no scores can be calculated.Here an example:#!/usr/bin/env python3
from transformers import AutoModelForSeq2SeqLM
import torch

model = AutoModelForSeq2SeqLM.from_pretrained('facebook/bart-large')

out = model.generate(torch.tensor([10 * [1]]), return_dict_in_generate=True, output_scores=True, max_length = 10)

print("len scores:", len(out.scores))  # should give 9Would you be interested in correcting the documentation in a PR for Transformers?
2021-02-07T02:15:54Z
Hi@patrickvonplaten,Thanks for your reply! That makes more sense. Sure, I can correct that in a PR.
2021-05-12T05:18:21Z
Hi@patrickvonplatenAny reason that thedecoder_start_token_idis concatenated to the beginning of the generated token ids?
2021-05-17T21:38:17Z
Hi@ad26kr, BART uses <bos> at the beginning of the decoder input to indicate the start of decoding. This is how the model pretrained (see Figure 1(c) of thepaper).
2021-05-18T03:03:44Z
Hi@KylieI know that several seq2seq models such as BART, T5 use some special tokens as the first input token to let the decoder start decoding. However, I can’t understand why ‘the generated text/tokens from the model.generate()’ includes the special tokens (in the beginning of the generated token ids)
2023-05-26T01:29:56Z
What does thescoresmeans?Thank you!