Learning in the Wild with Incremental Skeptical Gaussian Processes
  The ability to learn from human supervision is fundamental for personal
assistants and other interactive applications of AI. Two central challenges for
deploying interactive learners in the wild are the unreliable nature of the
supervision and the varying complexity of the prediction task. We address a
simple but representative setting, incremental classification in the wild,
where the supervision is noisy and the number of classes grows over time. In
order to tackle this task, we propose a redesign of skeptical learning centered
around Gaussian Processes (GPs). Skeptical learning is a recent interactive
strategy in which, if the machine is sufficiently confident that an example is
mislabeled, it asks the annotator to reconsider her feedback. In many cases,
this is often enough to obtain clean supervision. Our redesign, dubbed ISGP,
leverages the uncertainty estimates supplied by GPs to better allocate labeling
and contradiction queries, especially in the presence of noise. Our experiments
on synthetic and real-world data show that, as a result, while the original
formulation of skeptical learning produces over-confident models that can fail
completely in the wild, ISGP works well at varying levels of noise and as new
classes are observed.
