Factored Adaptation for Non-Stationary Reinforcement Learning
  Dealing with non-stationarity in environments (i.e., transition dynamics) and
objectives (i.e., reward functions) is a challenging problem that is crucial in
real-world applications of reinforcement learning (RL). Most existing
approaches only focus on families of stationary MDPs, in which the
non-stationarity is episodic, i.e., the change is only possible across
episodes. The few works that do consider non-stationarity without a specific
boundary, i.e., also allow for changes within an episode, model the changes
monolithically in a single shared embedding vector. In this paper, we propose
Factored Adaptation for Non-Stationary RL (FANS-RL), a factored adaption
approach that explicitly learns the individual latent change factors affecting
the transition dynamics and reward functions. FANS-RL learns jointly the
structure of a factored MDP and a factored representation of the time-varying
change factors, as well as the specific state components that they affect, via
a factored non-stationary variational autoencoder. Through this general
framework, we can consider general non-stationary scenarios with different
changing function types and changing frequency. Experimental results
demonstrate that FANS-RL outperforms existing approaches in terms of rewards,
compactness of the latent state representation and robustness to varying
degrees of non-stationarity.
