Exploiting Uncertainties from Ensemble Learners to Improve
  Decision-Making in Healthcare AI
  Ensemble learning is widely applied in Machine Learning (ML) to improve model
performance and to mitigate decision risks. In this approach, predictions from
a diverse set of learners are combined to obtain a joint decision. Recently,
various methods have been explored in literature for estimating decision
uncertainties using ensemble learning; however, determining which metrics are a
better fit for certain decision-making applications remains a challenging task.
In this paper, we study the following key research question in the selection of
uncertainty metrics: when does an uncertainty metric outperforms another? We
answer this question via a rigorous analysis of two commonly used uncertainty
metrics in ensemble learning, namely ensemble mean and ensemble variance. We
show that, under mild assumptions on the ensemble learners, ensemble mean is
preferable with respect to ensemble variance as an uncertainty metric for
decision making. We empirically validate our assumptions and theoretical
results via an extensive case study: the diagnosis of referable diabetic
retinopathy.
