Deja-vu: Double Feature Presentation and Iterated Loss in Deep
  Transformer Networks
  Deep acoustic models typically receive features in the first layer of the
network, and process increasingly abstract representations in the subsequent
layers. Here, we propose to feed the input features at multiple depths in the
acoustic model. As our motivation is to allow acoustic models to re-examine
their input features in light of partial hypotheses we introduce intermediate
model heads and loss function. We study this architecture in the context of
deep Transformer networks, and we use an attention mechanism over both the
previous layer activations and the input features. To train this model's
intermediate output hypothesis, we apply the objective function at each layer
right before feature re-use. We find that the use of such iterated loss
significantly improves performance by itself, as well as enabling input feature
re-use. We present results on both Librispeech, and a large scale video
dataset, with relative improvements of 10 - 20% for Librispeech and 3.2 - 13%
for videos.
