Frequency Principle in Deep Learning Beyond Gradient-descent-based
  Training
  Frequency perspective recently makes progress in understanding deep learning.
It has been widely verified in both empirical and theoretical studies that deep
neural networks (DNNs) often fit the target function from low to high
frequency, namely Frequency Principle (F-Principle). F-Principle sheds light on
the strength and the weakness of DNNs and inspires a series of subsequent
works, including theoretical studies, empirical studies and the design of
efficient DNN structures etc. Previous works examine the F-Principle in
gradient-descent-based training. It remains unclear whether
gradient-descent-based training is a necessary condition for the F-Principle.
In this paper, we show that the F-Principle exists stably in the training
process of DNNs with non-gradient-descent-based training, including
optimization algorithms with gradient information, such as conjugate gradient
and BFGS, and algorithms without gradient information, such as Powell's method
and Particle Swarm Optimization. These empirical studies show the universality
of the F-Principle and provide hints for further study of F-Principle.
