Implicit Regularization Properties of Variance Reduced Stochastic Mirror
  Descent
  In machine learning and statistical data analysis, we often run into
objective function that is a summation: the number of terms in the summation
possibly is equal to the sample size, which can be enormous. In such a setting,
the stochastic mirror descent (SMD) algorithm is a numerically efficient method
-- each iteration involving a very small subset of the data. The variance
reduction version of SMD (VRSMD) can further improve SMD by inducing faster
convergence. On the other hand, algorithms such as gradient descent and
stochastic gradient descent have the implicit regularization property that
leads to better performance in terms of the generalization errors. Little is
known on whether such a property holds for VRSMD. We prove here that the
discrete VRSMD estimator sequence converges to the minimum mirror interpolant
in the linear regression. This establishes the implicit regularization property
for VRSMD. As an application of the above result, we derive a model estimation
accuracy result in the setting when the true model is sparse. We use numerical
examples to illustrate the empirical power of VRSMD.
