Explicitizing an Implicit Bias of the Frequency Principle in Two-layer
  Neural Networks
  It remains a puzzle that why deep neural networks (DNNs), with more
parameters than samples, often generalize well. An attempt of understanding
this puzzle is to discover implicit biases underlying the training process of
DNNs, such as the Frequency Principle (F-Principle), i.e., DNNs often fit
target functions from low to high frequencies. Inspired by the F-Principle, we
propose an effective model of linear F-Principle (LFP) dynamics which
accurately predicts the learning results of two-layer ReLU neural networks
(NNs) of large widths. This LFP dynamics is rationalized by a linearized mean
field residual dynamics of NNs. Importantly, the long-time limit solution of
this LFP dynamics is equivalent to the solution of a constrained optimization
problem explicitly minimizing an FP-norm, in which higher frequencies of
feasible solutions are more heavily penalized. Using this optimization
formulation, an a priori estimate of the generalization error bound is
provided, revealing that a higher FP-norm of the target function increases the
generalization error. Overall, by explicitizing the implicit bias of the
F-Principle as an explicit penalty for two-layer NNs, our work makes a step
towards a quantitative understanding of the learning and generalization of
general DNNs.
