On the Proof of Global Convergence of Gradient Descent for Deep ReLU
  Networks with Linear Widths
  We give a simple proof for the global convergence of gradient descent in
training deep ReLU networks with the standard square loss, and show some of its
improvements over the state-of-the-art. In particular, while prior works
require all the hidden layers to be wide with width at least $\Omega(N^8)$ ($N$
being the number of training samples), we require a single wide layer of
linear, quadratic or cubic width depending on the type of initialization.
Unlike many recent proofs based on the Neural Tangent Kernel (NTK), our proof
need not track the evolution of the entire NTK matrix, or more generally, any
quantities related to the changes of activation patterns during training.
Instead, we only need to track the evolution of the output at the last hidden
layer, which can be done much more easily thanks to the Lipschitz property of
ReLU. Some highlights of our setting: (i) all the layers are trained with
standard gradient descent, (ii) the network has standard parameterization as
opposed to the NTK one, and (iii) the network has a single wide layer as
opposed to having all wide hidden layers as in most of NTK-related results.
