Temporal Variability in Implicit Online Learning
  In the setting of online learning, Implicit algorithms turn out to be highly
successful from a practical standpoint. However, the tightest regret analyses
only show marginal improvements over Online Mirror Descent. In this work, we
shed light on this behavior carrying out a careful regret analysis. We prove a
novel static regret bound that depends on the temporal variability of the
sequence of loss functions, a quantity which is often encountered when
considering dynamic competitors. We show, for example, that the regret can be
constant if the temporal variability is constant and the learning rate is tuned
appropriately, without the need of smooth losses. Moreover, we present an
adaptive algorithm that achieves this regret bound without prior knowledge of
the temporal variability and prove a matching lower bound. Finally, we validate
our theoretical findings on classification and regression datasets.
