The training response law explains how deep neural networks learn
  Deep neural network is the widely applied technology in this decade. In spite
of the fruitful applications, the mechanism behind that is still to be
elucidated. We study the learning process with a very simple supervised
learning encoding problem. As a result, we found a simple law, in the training
response, which describes neural tangent kernel. The response consists of a
power law like decay multiplied by a simple response kernel. We can construct a
simple mean-field dynamical model with the law, which explains how the network
learns. In the learning, the input space is split into sub-spaces along
competition between the kernels. With the iterated splits and the aging, the
network gets more complexity, but finally loses its plasticity.
