Tensor Programs II: Neural Tangent Kernel for Any Architecture
  We prove that a randomly initialized neural network of *any architecture* has
its Tangent Kernel (NTK) converge to a deterministic limit, as the network
widths tend to infinity. We demonstrate how to calculate this limit. In prior
literature, the heuristic study of neural network gradients often assumes every
weight matrix used in forward propagation is independent from its transpose
used in backpropagation (Schoenholz et al. 2017). This is known as the
*gradient independence assumption (GIA)*. We identify a commonly satisfied
condition, which we call *Simple GIA Check*, such that the NTK limit
calculation based on GIA is correct. Conversely, when Simple GIA Check fails,
we show GIA can result in wrong answers. Our material here presents the NTK
results of Yang (2019a) in a friendly manner and showcases the *tensor
programs* technique for understanding wide neural networks. We provide
reference implementations of infinite-width NTKs of recurrent neural network,
transformer, and batch normalization at https://github.com/thegregyang/NTK4A.
