The Stability and Accuracy Tradeoff Under Dataset Shift: A Causal
  Graphical Analysis
  Recent interest in dataset shift has produced many methods for finding
invariant distributions for prediction in new, unseen environments. However,
these methods consider different types of shifts and have been developed under
disparate frameworks, making it difficult to theoretically analyze how
solutions differ with respect to stability and accuracy. Taking a causal
graphical view, we use a flexible graphical representation to express various
types of dataset shifts. We show that all invariant distributions correspond to
a causal hierarchy of graphical operators which disable the edges in the graph
that are responsible for the shifts. The hierarchy provides a common
theoretical underpinning for understanding when and how stability to shifts can
be achieved, and in what ways stable distributions can differ. We use it to
establish conditions for minimax optimal performance across environments, and
derive new algorithms that find optimal stable distributions. Using this new
perspective, we empirically demonstrate that that there is a tradeoff between
minimax and average performance.
