Orthogonality Constrained Multi-Head Attention For Keyword Spotting
  Multi-head attention mechanism is capable of learning various representations
from sequential data while paying attention to different subsequences, e.g.,
word-pieces or syllables in a spoken word. From the subsequences, it retrieves
richer information than a single-head attention which only summarizes the whole
sequence into one context vector. However, a naive use of the multi-head
attention does not guarantee such richness as the attention heads may have
positional and representational redundancy. In this paper, we propose a
regularization technique for multi-head attention mechanism in an end-to-end
neural keyword spotting system. Augmenting regularization terms which penalize
positional and contextual non-orthogonality between the attention heads
encourages to output different representations from separate subsequences,
which in turn enables leveraging structured information without explicit
sequence models such as hidden Markov models. In addition, intra-head
contextual non-orthogonality regularization encourages each attention head to
have similar representations across keyword examples, which helps
classification by reducing feature variability. The experimental results
demonstrate that the proposed regularization technique significantly improves
the keyword spotting performance for the keyword "Hey Snapdragon".
