On the Global Convergence of Training Deep Linear ResNets
  We study the convergence of gradient descent (GD) and stochastic gradient
descent (SGD) for training $L$-hidden-layer linear residual networks (ResNets).
We prove that for training deep residual networks with certain linear
transformations at input and output layers, which are fixed throughout
training, both GD and SGD with zero initialization on all hidden weights can
converge to the global minimum of the training loss. Moreover, when
specializing to appropriate Gaussian random linear transformations, GD and SGD
provably optimize wide enough deep linear ResNets. Compared with the global
convergence result of GD for training standard deep linear networks (Du & Hu
2019), our condition on the neural network width is sharper by a factor of
$O(\kappa L)$, where $\kappa$ denotes the condition number of the covariance
matrix of the training data. We further propose a modified identity input and
output transformations, and show that a $(d+k)$-wide neural network is
sufficient to guarantee the global convergence of GD/SGD, where $d,k$ are the
input and output dimensions respectively.
