Online ICA: Understanding Global Dynamics of Nonconvex Optimization via
  Diffusion Processes
  Solving statistical learning problems often involves nonconvex optimization.
Despite the empirical success of nonconvex statistical optimization methods,
their global dynamics, especially convergence to the desirable local minima,
remain less well understood in theory. In this paper, we propose a new analytic
paradigm based on diffusion processes to characterize the global dynamics of
nonconvex statistical optimization. As a concrete example, we study stochastic
gradient descent (SGD) for the tensor decomposition formulation of independent
component analysis. In particular, we cast different phases of SGD into
diffusion processes, i.e., solutions to stochastic differential equations.
Initialized from an unstable equilibrium, the global dynamics of SGD transit
over three consecutive phases: (i) an unstable Ornstein-Uhlenbeck process
slowly departing from the initialization, (ii) the solution to an ordinary
differential equation, which quickly evolves towards the desirable local
minimum, and (iii) a stable Ornstein-Uhlenbeck process oscillating around the
desirable local minimum. Our proof techniques are based upon Stroock and
Varadhan's weak convergence of Markov chains to diffusion processes, which are
of independent interest.
