Rectified Factor Networks
  We propose rectified factor networks (RFNs) to efficiently construct very
sparse, non-linear, high-dimensional representations of the input. RFN models
identify rare and small events in the input, have a low interference between
code units, have a small reconstruction error, and explain the data covariance
structure. RFN learning is a generalized alternating minimization algorithm
derived from the posterior regularization method which enforces non-negative
and normalized posterior means. We proof convergence and correctness of the RFN
learning algorithm. On benchmarks, RFNs are compared to other unsupervised
methods like autoencoders, RBMs, factor analysis, ICA, and PCA. In contrast to
previous sparse coding methods, RFNs yield sparser codes, capture the data's
covariance structure more precisely, and have a significantly smaller
reconstruction error. We test RFNs as pretraining technique for deep networks
on different vision datasets, where RFNs were superior to RBMs and
autoencoders. On gene expression data from two pharmaceutical drug discovery
studies, RFNs detected small and rare gene modules that revealed highly
relevant new biological insights which were so far missed by other unsupervised
methods.
