Temporal Dependencies in Feature Importance for Time Series Predictions
  Explanation methods applied to sequential models for multivariate time series
prediction are receiving more attention in machine learning literature. While
current methods perform well at providing instance-wise explanations, they
struggle to efficiently and accurately make attributions over long periods of
time and with complex feature interactions. We propose WinIT, a framework for
evaluating feature importance in time series prediction settings by quantifying
the shift in predictive distribution over multiple instances in a windowed
setting. Comprehensive empirical evidence shows our method improves on the
previous state-of-the-art, FIT, by capturing temporal dependencies in feature
importance. We also demonstrate how the solution improves the appropriate
attribution of features within time steps, which existing interpretability
methods often fail to do. We compare with baselines on simulated and real-world
clinical data. WinIT achieves 2.47x better performance than FIT and other
feature importance methods on real-world clinical MIMIC-mortality task. The
code for this work is available at https://github.com/layer6ai-labs/WinIT.
