Effect of Depth and Width on Local Minima in Deep Learning
  In this paper, we analyze the effects of depth and width on the quality of
local minima, without strong over-parameterization and simplification
assumptions in the literature. Without any simplification assumption, for deep
nonlinear neural networks with the squared loss, we theoretically show that the
quality of local minima tends to improve towards the global minimum value as
depth and width increase. Furthermore, with a locally-induced structure on deep
nonlinear neural networks, the values of local minima of neural networks are
theoretically proven to be no worse than the globally optimal values of
corresponding classical machine learning models. We empirically support our
theoretical observation with a synthetic dataset as well as MNIST, CIFAR-10 and
SVHN datasets. When compared to previous studies with strong
over-parameterization assumptions, the results in this paper do not require
over-parameterization, and instead show the gradual effects of
over-parameterization as consequences of general results.
