Multiresolution Transformer Networks: Recurrence is Not Essential for
  Modeling Hierarchical Structure
  The architecture of Transformer is based entirely on self-attention, and has
been shown to outperform models that employ recurrence on sequence transduction
tasks such as machine translation. The superior performance of Transformer has
been attributed to propagating signals over shorter distances, between
positions in the input and the output, compared to the recurrent architectures.
We establish connections between the dynamics in Transformer and recurrent
networks to argue that several factors including gradient flow along an
ensemble of multiple weakly dependent paths play a paramount role in the
success of Transformer. We then leverage the dynamics to introduce {\em
Multiresolution Transformer Networks} as the first architecture that exploits
hierarchical structure in data via self-attention. Our models significantly
outperform state-of-the-art recurrent and hierarchical recurrent models on two
real-world datasets for query suggestion, namely, \aol and \amazon. In
particular, on AOL data, our model registers at least 20\% improvement on each
precision score, and over 25\% improvement on the BLEU score with respect to
the best performing recurrent model. We thus provide strong evidence that
recurrence is not essential for modeling hierarchical structure.
