Stochasticity of Deterministic Gradient Descent: Large Learning Rate for
  Multiscale Objective Function
  This article suggests that deterministic Gradient Descent, which does not use
any stochastic gradient approximation, can still exhibit stochastic behaviors.
In particular, it shows that if the objective function exhibit multiscale
behaviors, then in a large learning rate regime which only resolves the
macroscopic but not the microscopic details of the objective, the deterministic
GD dynamics can become chaotic and convergent not to a local minimizer but to a
statistical distribution. A sufficient condition is also established for
approximating this long-time statistical limit by a rescaled Gibbs
distribution. Both theoretical and numerical demonstrations are provided, and
the theoretical part relies on the construction of a stochastic map that uses
bounded noise (as opposed to discretized diffusions).
