Transformers are Deep Infinite-Dimensional Non-Mercer Binary Kernel
  Machines
  Despite their ubiquity in core AI fields like natural language processing,
the mechanics of deep attention-based neural networks like the Transformer
model are not fully understood. In this article, we present a new perspective
towards understanding how Transformers work. In particular, we show that the
"dot-product attention" that is the core of the Transformer's operation can be
characterized as a kernel learning method on a pair of Banach spaces. In
particular, the Transformer's kernel is characterized as having an infinite
feature dimension. Along the way we consider an extension of the standard
kernel learning problem to a binary setting, where data come from two input
domains and a response is defined for every cross-domain pair. We prove a new
representer theorem for these binary kernel machines with non-Mercer
(indefinite, asymmetric) kernels (implying that the functions learned are
elements of reproducing kernel Banach spaces rather than Hilbert spaces), and
also prove a new universal approximation theorem showing that the Transformer
calculation can learn any binary non-Mercer reproducing kernel Banach space
pair. We experiment with new kernels in Transformers, and obtain results that
suggest the infinite dimensionality of the standard Transformer kernel is
partially responsible for its performance. This paper's results provide a new
theoretical understanding of a very important but poorly understood model in
modern machine~learning.
