On the Neural Tangent Kernel of Deep Networks with Orthogonal
  Initialization
  The prevailing thinking is that orthogonal weights are crucial to enforcing
dynamical isometry and speeding up training. The increase in learning speed
that results from orthogonal initialization in linear networks has been
well-proven. However, while the same is believed to also hold for nonlinear
networks when the dynamical isometry condition is satisfied, the training
dynamics behind this contention have not been thoroughly explored. In this
work, we study the dynamics of ultra-wide networks across a range of
architectures, including Fully Connected Networks (FCNs) and Convolutional
Neural Networks (CNNs) with orthogonal initialization via neural tangent kernel
(NTK). Through a series of propositions and lemmas, we prove that two NTKs, one
corresponding to Gaussian weights and one to orthogonal weights, are equal when
the network width is infinite. Further, during training, the NTK of an
orthogonally-initialized infinite-width network should theoretically remain
constant. This suggests that the orthogonal initialization cannot speed up
training in the NTK (lazy training) regime, contrary to the prevailing
thoughts. In order to explore under what circumstances can orthogonality
accelerate training, we conduct a thorough empirical investigation outside the
NTK regime. We find that when the hyper-parameters are set to achieve a linear
regime in nonlinear activation, orthogonal initialization can improve the
learning speed with a large learning rate or large depth.
