Stopping criterion for active learning based on deterministic
  generalization bounds
  Active learning is a framework in which the learning machine can select the
samples to be used for training. This technique is promising, particularly when
the cost of data acquisition and labeling is high. In active learning,
determining the timing at which learning should be stopped is a critical issue.
In this study, we propose a criterion for automatically stopping active
learning. The proposed stopping criterion is based on the difference in the
expected generalization errors and hypothesis testing. We derive a novel upper
bound for the difference in expected generalization errors before and after
obtaining a new training datum based on PAC-Bayesian theory. Unlike ordinary
PAC-Bayesian bounds, though, the proposed bound is deterministic; hence, there
is no uncontrollable trade-off between the confidence and tightness of the
inequality. We combine the upper bound with a statistical test to derive a
stopping criterion for active learning. We demonstrate the effectiveness of the
proposed method via experiments with both artificial and real datasets.
