Overview frequency principle/spectral bias in deep learning
  Understanding deep learning is increasingly emergent as it penetrates more
and more into industry and science. In recent years, a research line from
Fourier analysis sheds lights into this magical "black box" by showing a
Frequency Principle (F-Principle or spectral bias) of the training behavior of
deep neural networks (DNNs) -- DNNs often fit functions from low to high
frequency during the training. The F-Principle is first demonstrated by
one-dimensional synthetic data followed by the verification in high-dimensional
real datasets. A series of works subsequently enhance the validity of the
F-Principle. This low-frequency implicit bias reveals the strength of neural
network in learning low-frequency functions as well as its deficiency in
learning high-frequency functions. Such understanding inspires the design of
DNN-based algorithms in practical problems, explains experimental phenomena
emerging in various scenarios, and further advances the study of deep learning
from the frequency perspective. Although incomplete, we provide an overview of
F-Principle and propose some open problems for future research.
