Universal Approximation of Input-Output Maps by Temporal Convolutional
  Nets
  There has been a recent shift in sequence-to-sequence modeling from recurrent
network architectures to convolutional network architectures due to
computational advantages in training and operation while still achieving
competitive performance. For systems having limited long-term temporal
dependencies, the approximation capability of recurrent networks is essentially
equivalent to that of temporal convolutional nets (TCNs). We prove that TCNs
can approximate a large class of input-output maps having approximately finite
memory to arbitrary error tolerance. Furthermore, we derive quantitative
approximation rates for deep ReLU TCNs in terms of the width and depth of the
network and modulus of continuity of the original input-output map, and apply
these results to input-output maps of systems that admit finite-dimensional
state-space realizations (i.e., recurrent models).
