What Kinds of Functions do Deep Neural Networks Learn? Insights from
  Variational Spline Theory
  We develop a variational framework to understand the properties of functions
learned by fitting deep neural networks with rectified linear unit activations
to data. We propose a new function space, which is reminiscent of classical
bounded variation-type spaces, that captures the compositional structure
associated with deep neural networks. We derive a representer theorem showing
that deep ReLU networks are solutions to regularized data fitting problems over
functions from this space. The function space consists of compositions of
functions from the Banach spaces of second-order bounded variation in the Radon
domain. These are Banach spaces with sparsity-promoting norms, giving insight
into the role of sparsity in deep neural networks. The neural network solutions
have skip connections and rank bounded weight matrices, providing new
theoretical support for these common architectural choices. The variational
problem we study can be recast as a finite-dimensional neural network training
problem with regularization schemes related to the notions of weight decay and
path-norm regularization. Finally, our analysis builds on techniques from
variational spline theory, providing new connections between deep neural
networks and splines.
