Training Neural Networks as Learning Data-adaptive Kernels: Provable
  Representation and Approximation Benefits
  Consider the problem: given the data pair $(\mathbf{x}, \mathbf{y})$ drawn
from a population with $f_*(x) = \mathbf{E}[\mathbf{y} | \mathbf{x} = x]$,
specify a neural network model and run gradient flow on the weights over time
until reaching any stationarity. How does $f_t$, the function computed by the
neural network at time $t$, relate to $f_*$, in terms of approximation and
representation? What are the provable benefits of the adaptive representation
by neural networks compared to the pre-specified fixed basis representation in
the classical nonparametric literature? We answer the above questions via a
dynamic reproducing kernel Hilbert space (RKHS) approach indexed by the
training process of neural networks. Firstly, we show that when reaching any
local stationarity, gradient flow learns an adaptive RKHS representation and
performs the global least-squares projection onto the adaptive RKHS,
simultaneously. Secondly, we prove that as the RKHS is data-adaptive and
task-specific, the residual for $f_*$ lies in a subspace that is potentially
much smaller than the orthogonal complement of the RKHS. The result formalizes
the representation and approximation benefits of neural networks. Lastly, we
show that the neural network function computed by gradient flow converges to
the kernel ridgeless regression with an adaptive kernel, in the limit of
vanishing regularization. The adaptive kernel viewpoint provides new angles of
studying the approximation, representation, generalization, and optimization
advantages of neural networks.
