Learning to Encode Position for Transformer with Continuous Dynamical
  Model
  We introduce a new way of learning to encode position information for
non-recurrent models, such as Transformer models. Unlike RNN and LSTM, which
contain inductive bias by loading the input tokens sequentially, non-recurrent
models are less sensitive to position. The main reason is that position
information among input units is not inherently encoded, i.e., the models are
permutation equivalent; this problem justifies why all of the existing models
are accompanied by a sinusoidal encoding/embedding layer at the input. However,
this solution has clear limitations: the sinusoidal encoding is not flexible
enough as it is manually designed and does not contain any learnable
parameters, whereas the position embedding restricts the maximum length of
input sequences. It is thus desirable to design a new position layer that
contains learnable parameters to adjust to different datasets and different
architectures. At the same time, we would also like the encodings to
extrapolate in accordance with the variable length of inputs. In our proposed
solution, we borrow from the recent Neural ODE approach, which may be viewed as
a versatile continuous version of a ResNet. This model is capable of modeling
many kinds of dynamical systems. We model the evolution of encoded results
along position index by such a dynamical system, thereby overcoming the above
limitations of existing methods. We evaluate our new position layers on a
variety of neural machine translation and language understanding tasks, the
experimental results show consistent improvements over the baselines.
