Tacotron2: tokens of pad value confuse decoder at inference
Closing. re-open if issue is still relevant.
**Describe your question**

When running inference on a batch with a Tacotron2 model, the decoder correctly ends sequences when none of the sequences in the batch have too many tokens with the ```pad``` (which is ```0``` here) value at the end - so inferencing is fine when the batch size is 1 as shown in the examples or the sequences in the batch happen to have very similar length.
However, when there are more than a few such tokens at the end, the decoder does not stop and reaches ```max_decoder_steps``` even for very large values of ```max_decoder_steps```, even on sequences of in-domain length, even for very low values of ```gate_threshold```.

Intuitively, I'd expect the model to have learned that any amount of ```pad``` tokens signal the end, especially since they're preceded by the ```eos``` token. The sequences are not longer than ones encountered during training, so there shouldn't be attention alignment problems, at least not every time as to guarantee this issue.

What could be going wrong? I suspect it may have something to do with either masking of loss during training and/or one of the dropouts being used, but I'm not sure how to tackle this.

Note that this issue is also replicated in the TTS example colab notebook - add as few as 5 zeroes to the end of your parsed input and see what happens. Note that in the example the pad value is equivalent to the token assigned to the whitespace character, which isn't the case in my application, although I'm not sure if that's of any significance given that the issue arises in both cases.

This issue is, in my view, fairly significant because it makes batch inferencing essentially impossible.
