A Correspondence Between Random Neural Networks and Statistical Field
  Theory
  A number of recent papers have provided evidence that practical design
questions about neural networks may be tackled theoretically by studying the
behavior of random networks. However, until now the tools available for
analyzing random neural networks have been relatively ad-hoc. In this work, we
show that the distribution of pre-activations in random neural networks can be
exactly mapped onto lattice models in statistical physics. We argue that
several previous investigations of stochastic networks actually studied a
particular factorial approximation to the full lattice model. For random linear
networks and random rectified linear networks we show that the corresponding
lattice models in the wide network limit may be systematically approximated by
a Gaussian distribution with covariance between the layers of the network. In
each case, the approximate distribution can be diagonalized by Fourier
transformation. We show that this approximation accurately describes the
results of numerical simulations of wide random neural networks. Finally, we
demonstrate that in each case the large scale behavior of the random networks
can be approximated by an effective field theory.
