A Theory of Generative ConvNet
  We show that a generative random field model, which we call generative
ConvNet, can be derived from the commonly used discriminative ConvNet, by
assuming a ConvNet for multi-category classification and assuming one of the
categories is a base category generated by a reference distribution. If we
further assume that the non-linearity in the ConvNet is Rectified Linear Unit
(ReLU) and the reference distribution is Gaussian white noise, then we obtain a
generative ConvNet model that is unique among energy-based models: The model is
piecewise Gaussian, and the means of the Gaussian pieces are defined by an
auto-encoder, where the filters in the bottom-up encoding become the basis
functions in the top-down decoding, and the binary activation variables
detected by the filters in the bottom-up convolution process become the
coefficients of the basis functions in the top-down deconvolution process. The
Langevin dynamics for sampling the generative ConvNet is driven by the
reconstruction error of this auto-encoder. The contrastive divergence learning
of the generative ConvNet reconstructs the training images by the auto-encoder.
The maximum likelihood learning algorithm can synthesize realistic natural
image patterns.
