Stochastic Neural Networks with Monotonic Activation Functions
  We propose a Laplace approximation that creates a stochastic unit from any
smooth monotonic activation function, using only Gaussian noise. This paper
investigates the application of this stochastic approximation in training a
family of Restricted Boltzmann Machines (RBM) that are closely linked to
Bregman divergences. This family, that we call exponential family RBM
(Exp-RBM), is a subset of the exponential family Harmoniums that expresses
family members through a choice of smooth monotonic non-linearity for each
neuron. Using contrastive divergence along with our Gaussian approximation, we
show that Exp-RBM can learn useful representations using novel stochastic
units.
