Focus on the present: a regularization method for the ASR source-target
  attention layer
  This paper introduces a novel method to diagnose the source-target attention
in state-of-the-art end-to-end speech recognition models with joint
connectionist temporal classification (CTC) and attention training. Our method
is based on the fact that both, CTC and source-target attention, are acting on
the same encoder representations. To understand the functionality of the
attention, CTC is applied to compute the token posteriors given the attention
outputs. We found that the source-target attention heads are able to predict
several tokens ahead of the current one. Inspired by the observation, a new
regularization method is proposed which leverages CTC to make source-target
attention more focused on the frames corresponding to the output token being
predicted by the decoder. Experiments reveal stable improvements up to 7\% and
13\% relatively with the proposed regularization on TED-LIUM 2 and LibriSpeech.
