Frequency Principle in Deep Learning with General Loss Functions and Its
  Potential Application
  Previous studies have shown that deep neural networks (DNNs) with common
settings often capture target functions from low to high frequency, which is
called Frequency Principle (F-Principle). It has also been shown that
F-Principle can provide an understanding to the often observed good
generalization ability of DNNs. However, previous studies focused on the loss
function of mean square error, while various loss functions are used in
practice. In this work, we show that the F-Principle holds for a general loss
function (e.g., mean square error, cross entropy, etc.). In addition, DNN's
F-Principle may be applied to develop numerical schemes for solving various
problems which would benefit from a fast converging of low frequency. As an
example of the potential usage of F-Principle, we apply DNN in solving
differential equations, in which conventional methods (e.g., Jacobi method) is
usually slow in solving problems due to the convergence from high to low
frequency.
