Lazy-MDPs: Towards Interpretable Reinforcement Learning by Learning When
  to Act
  Traditionally, Reinforcement Learning (RL) aims at deciding how to act
optimally for an artificial agent. We argue that deciding when to act is
equally important. As humans, we drift from default, instinctive or memorized
behaviors to focused, thought-out behaviors when required by the situation. To
enhance RL agents with this aptitude, we propose to augment the standard Markov
Decision Process and make a new mode of action available: being lazy, which
defers decision-making to a default policy. In addition, we penalize non-lazy
actions in order to encourage minimal effort and have agents focus on critical
decisions only. We name the resulting formalism lazy-MDPs. We study the
theoretical properties of lazy-MDPs, expressing value functions and
characterizing optimal solutions. Then we empirically demonstrate that policies
learned in lazy-MDPs generally come with a form of interpretability: by
construction, they show us the states where the agent takes control over the
default policy. We deem those states and corresponding actions important since
they explain the difference in performance between the default and the new,
lazy policy. With suboptimal policies as default (pretrained or random), we
observe that agents are able to get competitive performance in Atari games
while only taking control in a limited subset of states.
