Limitation of characterizing implicit regularization by data-independent
  functions
  In recent years, understanding the implicit regularization of neural networks
(NNs) has become a central task of deep learning theory. However, implicit
regularization is in itself not completely defined and well understood. In this
work, we make an attempt to mathematically define and study the implicit
regularization. Importantly, we explore the limitation of a common approach of
characterizing the implicit regularization by data-independent functions. We
propose two dynamical mechanisms, i.e., Two-point and One-point Overlapping
mechanisms, based on which we provide two recipes for producing classes of
one-hidden-neuron NNs that provably cannot be fully characterized by a type of
or all data-independent functions. Our results signify the profound
data-dependency of implicit regularization in general, inspiring us to study in
detail the data-dependency of NN implicit regularization in the future.
