Early Forecasting of Text Classification Accuracy and F-Measure with
  Active Learning
  When creating text classification systems, one of the major bottlenecks is
the annotation of training data. Active learning has been proposed to address
this bottleneck using stopping methods to minimize the cost of data annotation.
An important capability for improving the utility of stopping methods is to
effectively forecast the performance of the text classification models.
Forecasting can be done through the use of logarithmic models regressed on some
portion of the data as learning is progressing. A critical unexplored question
is what portion of the data is needed for accurate forecasting. There is a
tension, where it is desirable to use less data so that the forecast can be
made earlier, which is more useful, versus it being desirable to use more data,
so that the forecast can be more accurate. We find that when using active
learning it is even more important to generate forecasts earlier so as to make
them more useful and not waste annotation effort. We investigate the difference
in forecasting difficulty when using accuracy and F-measure as the text
classification system performance metrics and we find that F-measure is more
difficult to forecast. We conduct experiments on seven text classification
datasets in different semantic domains with different characteristics and with
three different base machine learning algorithms. We find that forecasting is
easiest for decision tree learning, moderate for Support Vector Machines, and
most difficult for neural networks.
