Lift Up and Act! Classifier Performance in Resource-Constrained
  Applications
  Classification tasks are common across many fields and applications where the
decision maker's action is limited by resource constraints. In direct marketing
only a subset of customers is contacted; scarce human resources limit the
number of interviews to the most promising job candidates; limited donated
organs are prioritized to those with best fit. In such scenarios, performance
measures such as the classification matrix, ROC analysis, and even ranking
metrics such as AUC measures outcomes different from the action of interest. At
the same time, gains and lift that do measure the relevant outcome are rarely
used by machine learners. In this paper we define resource-constrained
classifier performance as a task distinguished from classification and ranking.
We explain how gains and lift can lead to different algorithm choices and
discuss the effect of class distribution.
