On Optimality Conditions for Auto-Encoder Signal Recovery
  Auto-Encoders are unsupervised models that aim to learn patterns from
observed data by minimizing a reconstruction cost. The useful representations
learned are often found to be sparse and distributed. On the other hand,
compressed sensing and sparse coding assume a data generating process, where
the observed data is generated from some true latent signal source, and try to
recover the corresponding signal from measurements. Looking at auto-encoders
from this \textit{signal recovery perspective} enables us to have a more
coherent view of these techniques. In this paper, in particular, we show that
the \textit{true} hidden representation can be approximately recovered if the
weight matrices are highly incoherent with unit $ \ell^{2} $ row length and the
bias vectors takes the value (approximately) equal to the negative of the data
mean. The recovery also becomes more and more accurate as the sparsity in
hidden signals increases. Additionally, we empirically demonstrate that
auto-encoders are capable of recovering the data generating dictionary when
only data samples are given.
