Classifier Risk Estimation under Limited Labeling Resources
  In this paper we propose strategies for estimating performance of a
classifier when labels cannot be obtained for the whole test set. The number of
test instances which can be labeled is very small compared to the whole test
data size. The goal then is to obtain a precise estimate of classifier
performance using as little labeling resource as possible. Specifically, we try
to answer, how to select a subset of the large test set for labeling such that
the performance of a classifier estimated on this subset is as close as
possible to the one on the whole test set. We propose strategies based on
stratified sampling for selecting this subset. We show that these strategies
can reduce the variance in estimation of classifier accuracy by a significant
amount compared to simple random sampling (over 65% in several cases). Hence,
our proposed methods are much more precise compared to random sampling for
accuracy estimation under restricted labeling resources. The reduction in
number of samples required (compared to random sampling) to estimate the
classifier accuracy with only 1% error is high as 60% in some cases.
