Avoiding Side Effects By Considering Future Tasks
  Designing reward functions is difficult: the designer has to specify what to
do (what it means to complete the task) as well as what not to do (side effects
that should be avoided while completing the task). To alleviate the burden on
the reward designer, we propose an algorithm to automatically generate an
auxiliary reward function that penalizes side effects. This auxiliary objective
rewards the ability to complete possible future tasks, which decreases if the
agent causes side effects during the current task. The future task reward can
also give the agent an incentive to interfere with events in the environment
that make future tasks less achievable, such as irreversible actions by other
agents. To avoid this interference incentive, we introduce a baseline policy
that represents a default course of action (such as doing nothing), and use it
to filter out future tasks that are not achievable by default. We formally
define interference incentives and show that the future task approach with a
baseline policy avoids these incentives in the deterministic case. Using
gridworld environments that test for side effects and interference, we show
that our method avoids interference and is more effective for avoiding side
effects than the common approach of penalizing irreversible actions.
