Explanation-Aware Experience Replay in Rule-Dense Environments
  Human environments are often regulated by explicit and complex rulesets.
Integrating Reinforcement Learning (RL) agents into such environments motivates
the development of learning mechanisms that perform well in rule-dense and
exception-ridden environments such as autonomous driving on regulated roads. In
this paper, we propose a method for organising experience by means of
partitioning the experience buffer into clusters labelled on a per-explanation
basis. We present discrete and continuous navigation environments compatible
with modular rulesets and 9 learning tasks. For environments with explainable
rulesets, we convert rule-based explanations into case-based explanations by
allocating state-transitions into clusters labelled with explanations. This
allows us to sample experiences in a curricular and task-oriented manner,
focusing on the rarity, importance, and meaning of events. We label this
concept Explanation-Awareness (XA). We perform XA experience replay (XAER) with
intra and inter-cluster prioritisation, and introduce XA-compatible versions of
DQN, TD3, and SAC. Performance is consistently superior with XA versions of
those algorithms, compared to traditional Prioritised Experience Replay
baselines, indicating that explanation engineering can be used in lieu of
reward engineering for environments with explainable features.
