Variational Sparse Coding with Learned Thresholding
  Sparse coding strategies have been lauded for their parsimonious
representations of data that leverage low dimensional structure. However,
inference of these codes typically relies on an optimization procedure with
poor computational scaling in high-dimensional problems. For example, sparse
inference in the representations learned in the high-dimensional intermediary
layers of deep neural networks (DNNs) requires an iterative minimization to be
performed at each training step. As such, recent, quick methods in variational
inference have been proposed to infer sparse codes by learning a distribution
over the codes with a DNN. In this work, we propose a new approach to
variational sparse coding that allows us to learn sparse distributions by
thresholding samples, avoiding the use of problematic relaxations. We first
evaluate and analyze our method by training a linear generator, showing that it
has superior performance, statistical efficiency, and gradient estimation
compared to other sparse distributions. We then compare to a standard
variational autoencoder using a DNN generator on the Fashion MNIST and CelebA
datasets
