Rethinking clinical prediction: Why machine learning must consider year
  of care and feature aggregation
  Machine learning for healthcare often trains models on de-identified datasets
with randomly-shifted calendar dates, ignoring the fact that data were
generated under hospital operation practices that change over time. These
changing practices induce definitive changes in observed data which confound
evaluations which do not account for dates and limit the generalisability of
date-agnostic models. In this work, we establish the magnitude of this problem
on MIMIC, a public hospital dataset, and showcase a simple solution. We augment
MIMIC with the year in which care was provided and show that a model trained
using standard feature representations will significantly degrade in quality
over time. We find a deterioration of 0.3 AUC when evaluating mortality
prediction on data from 10 years later. We find a similar deterioration of 0.15
AUC for length-of-stay. In contrast, we demonstrate that clinically-oriented
aggregates of raw features significantly mitigate future deterioration. Our
suggested aggregated representations, when retrained yearly, have prediction
quality comparable to year-agnostic models.
