The Role of Neural Network Activation Functions
  A wide variety of activation functions have been proposed for neural
networks. The Rectified Linear Unit (ReLU) is especially popular today. There
are many practical reasons that motivate the use of the ReLU. This paper
provides new theoretical characterizations that support the use of the ReLU,
its variants such as the leaky ReLU, as well as other activation functions in
the case of univariate, single-hidden layer feedforward neural networks. Our
results also explain the importance of commonly used strategies in the design
and training of neural networks such as "weight decay" and "path-norm"
regularization, and provide a new justification for the use of "skip
connections" in network architectures. These new insights are obtained through
the lens of spline theory. In particular, we show how neural network training
problems are related to infinite-dimensional optimizations posed over Banach
spaces of functions whose solutions are well-known to be fractional and
polynomial splines, where the particular Banach space (which controls the order
of the spline) depends on the choice of activation function.
