A Bayesian Perspective of Convolutional Neural Networks through a
  Deconvolutional Generative Model
  Inspired by the success of Convolutional Neural Networks (CNNs) for
supervised prediction in images, we design the Deconvolutional Generative Model
(DGM), a new probabilistic generative model whose inference calculations
correspond to those in a given CNN architecture. The DGM uses a CNN to design
the prior distribution in the probabilistic model. Furthermore, the DGM
generates images from coarse to finer scales. It introduces a small set of
latent variables at each scale, and enforces dependencies among all the latent
variables via a conjugate prior distribution. This conjugate prior yields a new
regularizer based on paths rendered in the generative model for training
CNNs-the Rendering Path Normalization (RPN). We demonstrate that this
regularizer improves generalization, both in theory and in practice. In
addition, likelihood estimation in the DGM yields training losses for CNNs, and
inspired by this, we design a new loss termed as the Max-Min cross entropy
which outperforms the traditional cross-entropy loss for object classification.
The Max-Min cross entropy suggests a new deep network architecture, namely the
Max-Min network, which can learn from less labeled data while maintaining good
prediction performance. Our experiments demonstrate that the DGM with the RPN
and the Max-Min architecture exceeds or matches the-state-of-art on benchmarks
including SVHN, CIFAR10, and CIFAR100 for semi-supervised and supervised
learning tasks.
