Hybrid Orthogonal Projection and Estimation (HOPE): A New Framework to
  Probe and Learn Neural Networks
  In this paper, we propose a novel model for high-dimensional data, called the
Hybrid Orthogonal Projection and Estimation (HOPE) model, which combines a
linear orthogonal projection and a finite mixture model under a unified
generative modeling framework. The HOPE model itself can be learned
unsupervised from unlabelled data based on the maximum likelihood estimation as
well as discriminatively from labelled data. More interestingly, we have shown
the proposed HOPE models are closely related to neural networks (NNs) in a
sense that each hidden layer can be reformulated as a HOPE model. As a result,
the HOPE framework can be used as a novel tool to probe why and how NNs work,
more importantly, to learn NNs in either supervised or unsupervised ways. In
this work, we have investigated the HOPE framework to learn NNs for several
standard tasks, including image recognition on MNIST and speech recognition on
TIMIT. Experimental results have shown that the HOPE framework yields
significant performance gains over the current state-of-the-art methods in
various types of NN learning problems, including unsupervised feature learning,
supervised or semi-supervised learning.
