Leveraging Clinical Time-Series Data for Prediction: A Cautionary Tale
  In healthcare, patient risk stratification models are often learned using
time-series data extracted from electronic health records. When extracting data
for a clinical prediction task, several formulations exist, depending on how
one chooses the time of prediction and the prediction horizon. In this paper,
we show how the formulation can greatly impact both model performance and
clinical utility. Leveraging a publicly available ICU dataset, we consider two
clinical prediction tasks: in-hospital mortality, and hypokalemia. Through
these case studies, we demonstrate the necessity of evaluating models using an
outcome-independent reference point, since choosing the time of prediction
relative to the event can result in unrealistic performance. Further, an
outcome-independent scheme outperforms an outcome-dependent scheme on both
tasks (In-Hospital Mortality AUROC .882 vs. .831; Serum Potassium: AUROC .829
vs. .740) when evaluated on test sets that mimic real-world use.
