Performance measures for classification systems with rejection
  Classifiers with rejection are essential in real-world applications where
misclassifications and their effects are critical. However, if no problem
specific cost function is defined, there are no established measures to assess
the performance of such classifiers. We introduce a set of desired properties
for performance measures for classifiers with rejection, based on which we
propose a set of three performance measures for the evaluation of the
performance of classifiers with rejection that satisfy the desired properties.
The nonrejected accuracy measures the ability of the classifier to accurately
classify nonrejected samples; the classification quality measures the correct
decision making of the classifier with rejector; and the rejection quality
measures the ability to concentrate all misclassified samples onto the set of
rejected samples. From the measures, we derive the concept of relative
optimality that allows us to connect the measures to a family of cost functions
that take into account the trade-off between rejection and misclassification.
We illustrate the use of the proposed performance measures on classifiers with
rejection applied to synthetic and real-world data.
