Tackling benign nonconvexity with smoothing and stochastic gradients
  Non-convex optimization problems are ubiquitous in machine learning,
especially in Deep Learning. While such complex problems can often be
successfully optimized in practice by using stochastic gradient descent (SGD),
theoretical analysis cannot adequately explain this success. In particular, the
standard analyses do not show global convergence of SGD on non-convex
functions, and instead show convergence to stationary points (which can also be
local minima or saddle points). We identify a broad class of nonconvex
functions for which we can show that perturbed SGD (gradient descent perturbed
by stochastic noise -- covering SGD as a special case) converges to a global
minimum (or a neighborhood thereof), in contrast to gradient descent without
noise that can get stuck in local minima far from a global solution. For
example, on non-convex functions that are relatively close to a convex-like
(strongly convex or PL) function we show that SGD can converge linearly to a
global optimum.
