Non Asymptotic Bounds for Optimization via Online Multiplicative
  Stochastic Gradient Descent
  The gradient noise of Stochastic Gradient Descent (SGD) is considered to play
a key role in its properties (e.g. escaping low potential points and
regularization). Past research has indicated that the covariance of the SGD
error done via minibatching plays a critical role in determining its
regularization and escape from low potential points. It is however not much
explored how much the distribution of the error influences the behavior of the
algorithm. Motivated by some new research in this area, we prove universality
results by showing that noise classes that have the same mean and covariance
structure of SGD via minibatching have similar properties. We mainly consider
the Multiplicative Stochastic Gradient Descent (M-SGD) algorithm as introduced
by Wu et al., which has a much more general noise class than the SGD algorithm
done via minibatching. We establish nonasymptotic bounds for the M-SGD
algorithm mainly with respect to the Stochastic Differential Equation
corresponding to SGD via minibatching. We also show that the M-SGD error is
approximately a scaled Gaussian distribution with mean $0$ at any fixed point
of the M-SGD algorithm. We also establish bounds for the convergence of the
M-SGD algorithm in the strongly convex regime.
