Block Contextual MDPs for Continual Learning
  In reinforcement learning (RL), when defining a Markov Decision Process
(MDP), the environment dynamics is implicitly assumed to be stationary. This
assumption of stationarity, while simplifying, can be unrealistic in many
scenarios. In the continual reinforcement learning scenario, the sequence of
tasks is another source of nonstationarity. In this work, we propose to examine
this continual reinforcement learning setting through the block contextual MDP
(BC-MDP) framework, which enables us to relax the assumption of stationarity.
This framework challenges RL algorithms to handle both nonstationarity and rich
observation settings and, by additionally leveraging smoothness properties,
enables us to study generalization bounds for this setting. Finally, we take
inspiration from adaptive control to propose a novel algorithm that addresses
the challenges introduced by this more realistic BC-MDP setting, allows for
zero-shot adaptation at evaluation time, and achieves strong performance on
several nonstationary environments.
