Practical Active Learning with Model Selection for Small Data
  Active learning is of great interest for many practical applications,
especially in industry and the physical sciences, where there is a strong need
to minimize the number of costly experiments necessary to train predictive
models. However, there remain significant challenges for the adoption of active
learning methods in many practical applications. One important challenge is
that many methods assume a fixed model, where model hyperparameters are chosen
a priori. In practice, it is rarely true that a good model will be known in
advance. Existing methods for active learning with model selection typically
depend on a medium-sized labeling budget. In this work, we focus on the case of
having a very small labeling budget, on the order of a few dozen data points,
and develop a simple and fast method for practical active learning with model
selection. Our method is based on an underlying pool-based active learner for
binary classification using support vector classification with a radial basis
function kernel. First we show empirically that our method is able to find
hyperparameters that lead to the best performance compared to an oracle model
on less separable, difficult to classify datasets, and reasonable performance
on datasets that are more separable and easier to classify. Then, we
demonstrate that it is possible to refine our model selection method using a
weighted approach to trade-off between achieving optimal performance on
datasets that are easy to classify, versus datasets that are difficult to
classify, which can be tuned based on prior domain knowledge about the dataset.
