Knots in random neural networks
  The weights of a neural network are typically initialized at random, and one
can think of the functions produced by such a network as having been generated
by a prior over some function space. Studying random networks, then, is useful
for a Bayesian understanding of the network evolution in early stages of
training. In particular, one can investigate why neural networks with huge
numbers of parameters do not immediately overfit. We analyze the properties of
random scalar-input feed-forward rectified linear unit architectures, which are
random linear splines. With weights and biases sampled from certain common
distributions, empirical tests show that the number of knots in the spline
produced by the network is equal to the number of neurons, to very close
approximation. We describe our progress towards a completely analytic
explanation of this phenomenon. In particular, we show that random single-layer
neural networks are equivalent to integrated random walks with variable step
sizes. That each neuron produces one knot on average is equivalent to the
associated integrated random walk having one zero crossing on average. We
explore how properties of the integrated random walk, including the step sizes
and initial conditions, affect the number of crossings. The number of knots in
random neural networks can be related to the behavior of extreme learning
machines, but it also establishes a prior preventing optimizers from
immediately overfitting to noisy training data.
