Reward Tampering Problems and Solutions in Reinforcement Learning: A
  Causal Influence Diagram Perspective
  Can humans get arbitrarily capable reinforcement learning (RL) agents to do
their bidding? Or will sufficiently capable RL agents always find ways to
bypass their intended objectives by shortcutting their reward signal? This
question impacts how far RL can be scaled, and whether alternative paradigms
must be developed in order to build safe artificial general intelligence. In
this paper, we study when an RL agent has an instrumental goal to tamper with
its reward process, and describe design principles that prevent instrumental
goals for two different types of reward tampering (reward function tampering
and RF-input tampering). Combined, the design principles can prevent both types
of reward tampering from being instrumental goals. The analysis benefits from
causal influence diagrams to provide intuitive yet precise formalizations.
