Stochastic optimization under distributional drift
  We consider the problem of minimizing a convex function that is evolving
according to unknown and possibly stochastic dynamics, which may depend jointly
on time and on the decision variable itself. Such problems abound in the
machine learning and signal processing literature, under the names of concept
drift, stochastic tracking, and performative prediction. We provide novel
non-asymptotic convergence guarantees for stochastic algorithms with iterate
averaging, focusing on bounds valid both in expectation and with high
probability. The efficiency estimates we obtain clearly decouple the
contributions of optimization error, gradient noise, and time drift. Notably,
we show that the tracking efficiency of the proximal stochastic gradient method
depends only logarithmically on the initialization quality, when equipped with
a step-decay schedule. Numerical experiments illustrate our results.
