Online Continual Adaptation with Active Self-Training
  Models trained with offline data often suffer from continual distribution
shifts and expensive labeling in changing environments. This calls for a new
online learning paradigm where the learner can continually adapt to changing
environments with limited labels. In this paper, we propose a new online
setting -- Online Active Continual Adaptation, where the learner aims to
continually adapt to changing distributions using both unlabeled samples and
active queries of limited labels. To this end, we propose Online Self-Adaptive
Mirror Descent (OSAMD), which adopts an online teacher-student structure to
enable online self-training from unlabeled data, and a margin-based criterion
that decides whether to query the labels to track changing distributions.
Theoretically, we show that, in the separable case, OSAMD has an $O({T}^{2/3})$
dynamic regret bound under mild assumptions, which is aligned with the
$\Omega(T^{2/3})$ lower bound of online learning algorithms with full labels.
In the general case, we show a regret bound of $O({T}^{2/3} + \alpha^* T)$,
where $\alpha^*$ denotes the separability of domains and is usually small. Our
theoretical results show that OSAMD can fast adapt to changing environments
with active queries. Empirically, we demonstrate that OSAMD achieves favorable
regrets under changing environments with limited labels on both simulated and
real-world data, which corroborates our theoretical findings.
