A Simple and Effective Positional Encoding for Transformers
  Transformer models are permutation equivariant. To supply the order and type
information of the input tokens, position and segment embeddings are usually
added to the input. Recent works proposed variations of positional encodings
with relative position encodings achieving better performance. Our analysis
shows that the gain actually comes from moving positional information to
attention layer from the input. Motivated by this, we introduce Decoupled
Positional Attention for Transformers (DIET), a simple yet effective mechanism
to encode position and segment information into the Transformer models. The
proposed method has faster training and inference time, while achieving
competitive performance on GLUE, XTREME and WMT benchmarks. We further
generalize our method to long-range transformers and show performance gain.
