Avoiding Tampering Incentives in Deep RL via Decoupled Approval
  How can we design agents that pursue a given objective when all feedback
mechanisms are influenceable by the agent? Standard RL algorithms assume a
secure reward function, and can thus perform poorly in settings where agents
can tamper with the reward-generating mechanism. We present a principled
solution to the problem of learning from influenceable feedback, which combines
approval with a decoupled feedback collection procedure. For a natural class of
corruption functions, decoupled approval algorithms have aligned incentives
both at convergence and for their local updates. Empirically, they also scale
to complex 3D environments where tampering is possible.
