Improving Discrete Latent Representations With Differentiable
  Approximation Bridges
  Modern neural network training relies on piece-wise (sub-)differentiable
functions in order to use backpropagation to update model parameters. In this
work, we introduce a novel method to allow simple non-differentiable functions
at intermediary layers of deep neural networks. We do so by training with a
differentiable approximation bridge (DAB) neural network which approximates the
non-differentiable forward function and provides gradient updates during
backpropagation. We present strong empirical results (performing over 600
experiments) in four different domains: unsupervised (image) representation
learning, variational (image) density estimation, image classification, and
sequence sorting to demonstrate that our proposed method improves state of the
art performance. We demonstrate that training with DAB aided discrete
non-differentiable functions improves image reconstruction quality and
posterior linear separability by 10% against the Gumbel-Softmax relaxed
estimator [37, 26] as well as providing a 9% improvement in the test
variational lower bound in comparison to the state of the art RELAX [16]
discrete estimator. We also observe an accuracy improvement of 77% in neural
sequence sorting and a 25% improvement against the straight-through estimator
[5] in an image classification setting. The DAB network is not used for
inference and expands the class of functions that are usable in neural
networks.
