Finite size corrections for neural network Gaussian processes
  There has been a recent surge of interest in modeling neural networks (NNs)
as Gaussian processes. In the limit of a NN of infinite width the NN becomes
equivalent to a Gaussian process. Here we demonstrate that for an ensemble of
large, finite, fully connected networks with a single hidden layer the
distribution of outputs at initialization is well described by a Gaussian
perturbed by the fourth Hermite polynomial for weights drawn from a symmetric
distribution. We show that the scale of the perturbation is inversely
proportional to the number of units in the NN and that higher order terms decay
more rapidly, thereby recovering the Edgeworth expansion. We conclude by
observing that understanding how this perturbation changes under training would
reveal the regimes in which the Gaussian process framework is valid to model NN
behavior.
