Non-Vacuous Generalisation Bounds for Shallow Neural Networks
  We focus on a specific class of shallow neural networks with a single hidden
layer, namely those with $L_2$-normalised data and either a sigmoid-shaped
Gaussian error function ("erf") activation or a Gaussian Error Linear Unit
(GELU) activation. For these networks, we derive new generalisation bounds
through the PAC-Bayesian theory; unlike most existing such bounds they apply to
neural networks with deterministic rather than randomised parameters. Our
bounds are empirically non-vacuous when the network is trained with vanilla
stochastic gradient descent on MNIST and Fashion-MNIST.
