An Upper Limit of Decaying Rate with Respect to Frequency in Deep Neural
  Network
  Deep neural network (DNN) usually learns the target function from low to high
frequency, which is called frequency principle or spectral bias. This frequency
principle sheds light on a high-frequency curse of DNNs -- difficult to learn
high-frequency information. Inspired by the frequency principle, a series of
works are devoted to develop algorithms for overcoming the high-frequency
curse. A natural question arises: what is the upper limit of the decaying rate
w.r.t. frequency when one trains a DNN? In this work, our theory, confirmed by
numerical experiments, suggests that there is a critical decaying rate w.r.t.
frequency in DNN training. Below the upper limit of the decaying rate, the DNN
interpolates the training data by a function with a certain regularity.
However, above the upper limit, the DNN interpolates the training data by a
trivial function, i.e., a function is only non-zero at training data points.
Our results indicate a better way to overcome the high-frequency curse is to
design a proper pre-condition approach to shift high-frequency information to
low-frequency one, which coincides with several previous developed algorithms
for fast learning high-frequency information. More importantly, this work
rigorously proves that the high-frequency curse is an intrinsic difficulty of
DNNs.
