REALab: An Embedded Perspective on Tampering
  This paper describes REALab, a platform for embedded agency research in
reinforcement learning (RL). REALab is designed to model the structure of
tampering problems that may arise in real-world deployments of RL. Standard
Markov Decision Process (MDP) formulations of RL and simulated environments
mirroring the MDP structure assume secure access to feedback (e.g., rewards).
This may be unrealistic in settings where agents are embedded and can corrupt
the processes producing feedback (e.g., human supervisors, or an implemented
reward function). We describe an alternative Corrupt Feedback MDP formulation
and the REALab environment platform, which both avoid the secure feedback
assumption. We hope the design of REALab provides a useful perspective on
tampering problems, and that the platform may serve as a unit test for the
presence of tampering incentives in RL agent designs.
