Questions about the shape of T5 logits
The shape of logits in output is (batch,sequence_length,vocab_size). I don’t understand the sequence_length part. I thought decoder should predict one word at a time and the logits should be (batch,vocab_size) .Thank you in advance for any replies!
2021-09-22T09:32:19Z
Hi,Yes, but you always have a sequence length dimension. At the start of generation, we give the decoder start token to the T5 decoder.Suppose you have trained a T5 model to translate language from English to French, and that we now want to test it on the English sentence “Welcome to Paris”. In that case, the T5 encoder will first encode this sentence, so we getlast_hidden_statesof the encoder of shape(batch_size, sequence_length, hidden_size). As we only have a single sentence here,batch_size= 1, for simplicity let’s assume that every word is a token and we don’t add special tokens, sosequence_length= 3, and thehidden_sizeof a T5 base model is 768. So the output of the encoder is of shape (1, 3, 768).Next, we have the decoder. We first give it theconfig.decoder_start_token_id, which for T5 is equal to 0 (i.e. the id of the pad token =<pad>). This will be our only token at the beginning, hence sequence length = 1. So what we give as input (assuming we turned the decoder start token id into a vector) to the decoder is of shape(batch_size, sequence_length, hidden_size)= (1, 1, 768), and it will output the scores for each of the tokens of the vocabulary, hence shape(batch_size, sequence_length, vocab_size)= (1, 1, 32100). This will indicate which token T5 thinks will follow the pad token (so ideally it should output “Bienvenue”).Next, we give<pad> Bienvenueas input to the decoder, so now our sequence length is 2. Assuming we have turned both tokens into a vector, the input to the decoder is now of shape (1, 2, 768). It will output a tensor of shape (1, 2, 32100). We are only interested in the logits for the last token, and we will take that as the prediction for the next token (ideally it should output “à”).
2021-09-22T11:24:07Z
Thank you for replying. When I run the example in T5’doc, I get a logits with size(1,7,32128).WeChat220f3931580f22e83edb96409cd523ef1920×620 71.4 KBDoes this mean that the decoder has forwarded 7 times during training(with beam search, topp sampling or greedy search)? If so, where does this happen in the source code?I haven’t found it. I know it will forward 7 times when calling model.generate but not training or computing loss.Thanks again for your time!
2021-09-22T11:59:17Z
Does this mean that the decoder has forwarded 7 times during training(with beam search, topp sampling or greedy search)?No. During training, each training example is only forwarded once. We compare the outputs of the model (i.e.logits) to thelabels, for all token positions.Only at inference time, the generation happens sequentially (using.generate()), because then we feed the predictions of the model as input to the following
2021-09-23T03:24:18Z
I think I understand. For training, the decoder input is right shifted from labels whose sequence lengthis 7. For inference, the decoder input is a cls token which need to forward several times.Thank you for your time!