The Surprising Simplicity of the Early-Time Learning Dynamics of Neural
  Networks
  Modern neural networks are often regarded as complex black-box functions
whose behavior is difficult to understand owing to their nonlinear dependence
on the data and the nonconvexity in their loss landscapes. In this work, we
show that these common perceptions can be completely false in the early phase
of learning. In particular, we formally prove that, for a class of well-behaved
input distributions, the early-time learning dynamics of a two-layer
fully-connected neural network can be mimicked by training a simple linear
model on the inputs. We additionally argue that this surprising simplicity can
persist in networks with more layers and with convolutional architecture, which
we verify empirically. Key to our analysis is to bound the spectral norm of the
difference between the Neural Tangent Kernel (NTK) at initialization and an
affine transform of the data kernel; however, unlike many previous results
utilizing the NTK, we do not require the network to have disproportionately
large width, and the network is allowed to escape the kernel regime later in
training.
