Ridge Regression with Over-Parametrized Two-Layer Networks Converge to
  Ridgelet Spectrum
  Characterization of local minima draws much attention in theoretical studies
of deep learning. In this study, we investigate the distribution of parameters
in an over-parametrized finite neural network trained by ridge regularized
empirical square risk minimization (RERM). We develop a new theory of ridgelet
transform, a wavelet-like integral transform that provides a powerful and
general framework for the theoretical study of neural networks involving not
only the ReLU but general activation functions. We show that the distribution
of the parameters converges to a spectrum of the ridgelet transform. This
result provides a new insight into the characterization of the local minima of
neural networks, and the theoretical background of an inductive bias theory
based on lazy regimes. We confirm the visual resemblance between the parameter
distribution trained by SGD, and the ridgelet spectrum calculated by numerical
integration through numerical experiments with finite models.
