Depthwise Separable Convolutions for Neural Machine Translation
  Depthwise separable convolutions reduce the number of parameters and
computation used in convolutional operations while increasing representational
efficiency. They have been shown to be successful in image classification
models, both in obtaining better models than previously possible for a given
parameter count (the Xception architecture) and considerably reducing the
number of parameters required to perform at a given level (the MobileNets
family of architectures). Recently, convolutional sequence-to-sequence networks
have been applied to machine translation tasks with good results. In this work,
we study how depthwise separable convolutions can be applied to neural machine
translation. We introduce a new architecture inspired by Xception and ByteNet,
called SliceNet, which enables a significant reduction of the parameter count
and amount of computation needed to obtain results like ByteNet, and, with a
similar parameter count, achieves new state-of-the-art results. In addition to
showing that depthwise separable convolutions perform well for machine
translation, we investigate the architectural changes that they enable: we
observe that thanks to depthwise separability, we can increase the length of
convolution windows, removing the need for filter dilation. We also introduce a
new "super-separable" convolution operation that further reduces the number of
parameters and computational cost for obtaining state-of-the-art results.
