Generative learning for deep networks
  Learning, taking into account full distribution of the data, referred to as
generative, is not feasible with deep neural networks (DNNs) because they model
only the conditional distribution of the outputs given the inputs. Current
solutions are either based on joint probability models facing difficult
estimation problems or learn two separate networks, mapping inputs to outputs
(recognition) and vice-versa (generation). We propose an intermediate approach.
First, we show that forward computation in DNNs with logistic sigmoid
activations corresponds to a simplified approximate Bayesian inference in a
directed probabilistic multi-layer model. This connection allows to interpret
DNN as a probabilistic model of the output and all hidden units given the
input. Second, we propose that in order for the recognition and generation
networks to be more consistent with the joint model of the data, weights of the
recognition and generator network should be related by transposition. We
demonstrate in a tentative experiment that such a coupled pair can be learned
generatively, modelling the full distribution of the data, and has enough
capacity to perform well in both recognition and generation.
