Neural Tangent Kernel Beyond the Infinite-Width Limit: Effects of Depth
  and Initialization
  Neural Tangent Kernel (NTK) is widely used to analyze overparametrized neural
networks due to the famous result by (Jacot et al., 2018): in the
infinite-width limit, the NTK is deterministic and constant during training.
However, this result cannot explain the behavior of deep networks, since it
generally does not hold if depth and width tend to infinity simultaneously. In
this paper, we study the NTK of fully-connected ReLU networks with depth
comparable to width. We prove that the NTK properties depend significantly on
the depth-to-width ratio and the distribution of parameters at initialization.
In fact, our results indicate the importance of the three phases in the
hyperparameter space identified in (Poole et al., 2016): ordered, chaotic and
the edge of chaos (EOC). We derive exact expressions for the NTK dispersion in
the infinite-depth-and-width limit in all three phases and conclude that the
NTK variability grows exponentially with depth at the EOC and in the chaotic
phase but not in the ordered phase. We also show that the NTK of deep networks
may stay constant during training only in the ordered phase and discuss how the
structure of the NTK matrix changes during training.
