Reward Reports for Reinforcement Learning
  The desire to build good systems in the face of complex societal effects
requires a dynamic approach towards equity and access. Recent approaches to
machine learning (ML) documentation have demonstrated the promise of discursive
frameworks for deliberation about these complexities. However, these
developments have been grounded in a static ML paradigm, leaving the role of
feedback and post-deployment performance unexamined. Meanwhile, recent work in
reinforcement learning design has shown that the effects of optimization
objectives on the resultant system behavior can be wide-ranging and
unpredictable. In this paper we sketch a framework for documenting deployed
learning systems, which we call Reward Reports. Taking inspiration from various
contributions to the technical literature on reinforcement learning, we outline
Reward Reports as living documents that track updates to design choices and
assumptions behind what a particular automated system is optimizing for. They
are intended to track dynamic phenomena arising from system deployment, rather
than merely static properties of models or data. After presenting the elements
of a Reward Report, we provide three examples: DeepMind's MuZero, MovieLens,
and a hypothetical deployment of a Project Flow traffic control policy.
