Lifelong Incremental Reinforcement Learning with Online Bayesian
  Inference
  A central capability of a long-lived reinforcement learning (RL) agent is to
incrementally adapt its behavior as its environment changes, and to
incrementally build upon previous experiences to facilitate future learning in
real-world scenarios. In this paper, we propose LifeLong Incremental
Reinforcement Learning (LLIRL), a new incremental algorithm for efficient
lifelong adaptation to dynamic environments. We develop and maintain a library
that contains an infinite mixture of parameterized environment models, which is
equivalent to clustering environment parameters in a latent space. The prior
distribution over the mixture is formulated as a Chinese restaurant process
(CRP), which incrementally instantiates new environment models without any
external information to signal environmental changes in advance. During
lifelong learning, we employ the expectation maximization (EM) algorithm with
online Bayesian inference to update the mixture in a fully incremental manner.
In EM, the E-step involves estimating the posterior expectation of
environment-to-cluster assignments, while the M-step updates the environment
parameters for future learning. This method allows for all environment models
to be adapted as necessary, with new models instantiated for environmental
changes and old models retrieved when previously seen environments are
encountered again. Experiments demonstrate that LLIRL outperforms relevant
existing methods, and enables effective incremental adaptation to various
dynamic environments for lifelong learning.
