Population and Empirical PR Curves for Assessment of Ranking Algorithms
  The ROC curve is widely used to assess the quality of
prediction/classification/ranking algorithms, and its properties have been
extensively studied. The precision-recall (PR) curve has become the de facto
replacement for the ROC curve in the presence of imbalance, namely where one
class is far more likely than the other class. While the PR and ROC curves tend
to be used interchangeably, they have some very different properties.
Properties of the PR curve are the focus of this paper. We consider: (1)
population PR curves, where complete distributional assumptions are specified
for scores from both classes; and (2) empirical estimators of the PR curve,
where we observe scores and no distributional assumptions are made. The
properties have direct consequence on how the PR curve should, and should not,
be used. For example, the empirical PR curve is not consistent when scores in
the class of primary interest come from discrete distributions. On the other
hand, a normal approximation can fit quite well for points on the empirical PR
curve from continuously-defined scores, but convergence can be heavily
influenced by the distributional setting, the amount of imbalance, and the
point of interest on the PR curve.
