XCS Classifier System with Experience Replay
  XCS constitutes the most deeply investigated classifier system today. It
bears strong potentials and comes with inherent capabilities for mastering a
variety of different learning tasks. Besides outstanding successes in various
classification and regression tasks, XCS also proved very effective in certain
multi-step environments from the domain of reinforcement learning. Especially
in the latter domain, recent advances have been mainly driven by algorithms
which model their policies based on deep neural networks -- among which the
Deep-Q-Network (DQN) is a prominent representative. Experience Replay (ER)
constitutes one of the crucial factors for the DQN's successes, since it
facilitates stabilized training of the neural network-based Q-function
approximators. Surprisingly, XCS barely takes advantage of similar mechanisms
that leverage stored raw experiences encountered so far. To bridge this gap,
this paper investigates the benefits of extending XCS with ER. On the one hand,
we demonstrate that for single-step tasks ER bears massive potential for
improvements in terms of sample efficiency. On the shady side, however, we
reveal that the use of ER might further aggravate well-studied issues not yet
solved for XCS when applied to sequential decision problems demanding for
long-action-chains.
