A Probabilistic Framework for Nonlinearities in Stochastic Neural
  Networks
  We present a probabilistic framework for nonlinearities, based on doubly
truncated Gaussian distributions. By setting the truncation points
appropriately, we are able to generate various types of nonlinearities within a
unified framework, including sigmoid, tanh and ReLU, the most commonly used
nonlinearities in neural networks. The framework readily integrates into
existing stochastic neural networks (with hidden units characterized as random
variables), allowing one for the first time to learn the nonlinearities
alongside model weights in these networks. Extensive experiments demonstrate
the performance improvements brought about by the proposed framework when
integrated with the restricted Boltzmann machine (RBM), temporal RBM and the
truncated Gaussian graphical model (TGGM).
