Avoiding C-hacking when evaluating survival distribution predictions
  with discrimination measures
  In this paper we consider how to evaluate survival distribution predictions
with measures of discrimination. This is a non-trivial problem as
discrimination measures are the most commonly used in survival analysis and yet
there is no clear method to derive a risk prediction from a distribution
prediction. We survey methods proposed in literature and software and consider
their respective advantages and disadvantages. Whilst distributions are
frequently evaluated by discrimination measures, we find that the method for
doing so is rarely described in the literature and often leads to unfair
comparisons. We find that the most robust method of reducing a distribution to
a risk is to sum over the predicted cumulative hazard. We recommend that
machine learning survival analysis software implements clear transformations
between distribution and risk predictions in order to allow more transparent
and accessible model evaluation. The code used in the final experiment is
available at https://github.com/RaphaelS1/distribution_discrimination.
