Revisiting Landscape Analysis in Deep Neural Networks: Eliminating
  Decreasing Paths to Infinity
  Traditional landscape analysis of deep neural networks aims to show that no
sub-optimal local minima exist in some appropriate sense. From this, one may be
tempted to conclude that descent algorithms which escape saddle points will
reach a good local minimum. However, basic optimization theory tell us that it
is also possible for a descent algorithm to diverge to infinity if there are
paths leading to infinity, along which the loss function decreases. It is not
clear whether for non-linear neural networks there exists one setting that no
bad local-min and no decreasing paths to infinity can be simultaneously
achieved. In this paper, we give the first positive answer to this question.
More specifically, for a large class of over-parameterized deep neural networks
with appropriate regularizers, the loss function has no bad local minima and no
decreasing paths to infinity. The key mathematical trick is to show that the
set of regularizers which may be undesirable can be viewed as the image of a
Lipschitz continuous mapping from a lower-dimensional Euclidean space to a
higher-dimensional Euclidean space, and thus has zero measure.
