Stochastic Neural Networks with Infinite Width are Deterministic
  This work theoretically studies stochastic neural networks, a main type of
neural network in use. We prove that as the width of an optimized stochastic
neural network tends to infinity, its predictive variance on the training set
decreases to zero. Our theory justifies the common intuition that adding
stochasticity to the model can help regularize the model by introducing an
averaging effect. Two common examples that our theory can be relevant to are
neural networks with dropout and Bayesian latent variable models in a special
limit. Our result thus helps better understand how stochasticity affects the
learning of neural networks and potentially design better architectures for
practical problems.
