Online Active Model Selection for Pre-trained Classifiers
  Given $k$ pre-trained classifiers and a stream of unlabeled data examples,
how can we actively decide when to query a label so that we can distinguish the
best model from the rest while making a small number of queries? Answering this
question has a profound impact on a range of practical scenarios. In this work,
we design an online selective sampling approach that actively selects
informative examples to label and outputs the best model with high probability
at any round. Our algorithm can be used for online prediction tasks for both
adversarial and stochastic streams. We establish several theoretical guarantees
for our algorithm and extensively demonstrate its effectiveness in our
experimental studies.
