Out-of-Distribution Dynamics Detection: RL-Relevant Benchmarks and
  Results
  We study the problem of out-of-distribution dynamics (OODD) detection, which
involves detecting when the dynamics of a temporal process change compared to
the training-distribution dynamics. This is relevant to applications in
control, reinforcement learning (RL), and multi-variate time-series, where
changes to test time dynamics can impact the performance of learning
controllers/predictors in unknown ways. This problem is particularly important
in the context of deep RL, where learned controllers often overfit to the
training environment. Currently, however, there is a lack of established OODD
benchmarks for the types of environments commonly used in RL research. Our
first contribution is to design a set of OODD benchmarks derived from common RL
environments with varying types and intensities of OODD. Our second
contribution is to design a strong OODD baseline approach based on recurrent
implicit quantile network (RIQN), which monitors autoregressive prediction
errors for OODD detection. In addition to RIQN, we introduce and test three
other simpler baselines. Our final contribution is to evaluate our baseline
approaches on the benchmarks to provide results for future comparison.
