A Deep Conditioning Treatment of Neural Networks
  We study the role of depth in training randomly initialized overparameterized
neural networks. We give a general result showing that depth improves
trainability of neural networks by improving the conditioning of certain kernel
matrices of the input data. This result holds for arbitrary non-linear
activation functions under a certain normalization. We provide versions of the
result that hold for training just the top layer of the neural network, as well
as for training all layers, via the neural tangent kernel. As applications of
these general results, we provide a generalization of the results of Das et al.
(2019) showing that learnability of deep random neural networks with a large
class of non-linear activations degrades exponentially with depth. We also show
how benign overfitting can occur in deep neural networks via the results of
Bartlett et al. (2019b). We also give experimental evidence that normalized
versions of ReLU are a viable alternative to more complex operations like Batch
Normalization in training deep neural networks.
