A Study of Continual Learning Methods for Q-Learning
  We present an empirical study on the use of continual learning (CL) methods
in a reinforcement learning (RL) scenario, which, to the best of our knowledge,
has not been described before. CL is a very active recent research topic
concerned with machine learning under non-stationary data distributions.
Although this naturally applies to RL, the use of dedicated CL methods is still
uncommon. This may be due to the fact that CL methods often assume a
decomposition of CL problems into disjoint sub-tasks of stationary
distribution, that the onset of these sub-tasks is known, and that sub-tasks
are non-contradictory. In this study, we perform an empirical comparison of
selected CL methods in a RL problem where a physically simulated robot must
follow a racetrack by vision. In order to make CL methods applicable, we
restrict the RL setting and introduce non-conflicting subtasks of known onset,
which are however not disjoint and whose distribution, from the learner's point
of view, is still non-stationary. Our results show that dedicated CL methods
can significantly improve learning when compared to the baseline technique of
"experience replay".
