When Does Uncertainty Matter?: Understanding the Impact of Predictive
  Uncertainty in ML Assisted Decision Making
  As machine learning (ML) models are increasingly being employed to assist
human decision makers, it becomes critical to provide these decision makers
with relevant inputs which can help them decide if and how to incorporate model
predictions into their decision making. For instance, communicating the
uncertainty associated with model predictions could potentially be helpful in
this regard. However, there is little to no research that systematically
explores if and how conveying predictive uncertainty impacts decision making.
In this work, we carry out user studies to systematically assess how people
respond to different types of predictive uncertainty i.e., posterior predictive
distributions with different shapes and variances, in the context of ML
assisted decision making. To the best of our knowledge, this work marks one of
the first attempts at studying this question. Our results demonstrate that
people are more likely to agree with a model prediction when they observe the
corresponding uncertainty associated with the prediction. This finding holds
regardless of the properties (shape or variance) of predictive uncertainty
(posterior predictive distribution), suggesting that uncertainty is an
effective tool for persuading humans to agree with model predictions.
Furthermore, we also find that other factors such as domain expertise and
familiarity with ML also play a role in determining how someone interprets and
incorporates predictive uncertainty into their decision making.
