Why Did This Model Forecast This Future? Closed-Form Temporal Saliency
  Towards Causal Explanations of Probabilistic Forecasts
  Forecasting tasks surrounding the dynamics of low-level human behavior are of
significance to multiple research domains. In such settings, methods for
explaining specific forecasts can enable domain experts to gain insights into
the predictive relationships between behaviors. In this work, we introduce and
address the following question: given a probabilistic forecasting model how can
we identify observed windows that the model considers salient when making its
forecasts? We build upon a general definition of information-theoretic saliency
grounded in human perception and extend it to forecasting settings by
leveraging a crucial attribute of the domain: a single observation can result
in multiple valid futures. We propose to express the saliency of an observed
window in terms of the differential entropy of the resulting predicted future
distribution. In contrast to existing methods that either require explicit
training of the saliency mechanism or access to the internal states of the
forecasting model, we obtain a closed-form solution for the saliency map for
commonly used density functions in probabilistic forecasting. We empirically
demonstrate how our framework can recover salient observed windows from head
pose features for the sample task of speaking-turn forecasting using a
synthesized conversation dataset.
