Choices, Risks, and Reward Reports: Charting Public Policy for
  Reinforcement Learning Systems
  In the long term, reinforcement learning (RL) is considered by many AI
theorists to be the most promising path to artificial general intelligence.
This places RL practitioners in a position to design systems that have never
existed before and lack prior documentation in law and policy. Public agencies
could intervene on complex dynamics that were previously too opaque to
deliberate about, and long-held policy ambitions would finally be made
tractable. In this whitepaper we illustrate this potential and how it might be
technically enacted in the domains of energy infrastructure, social media
recommender systems, and transportation. Alongside these unprecedented
interventions come new forms of risk that exacerbate the harms already
generated by standard machine learning tools. We correspondingly present a new
typology of risks arising from RL design choices, falling under four
categories: scoping the horizon, defining rewards, pruning information, and
training multiple agents. Rather than allowing RL systems to unilaterally
reshape human domains, policymakers need new mechanisms for the rule of reason,
foreseeability, and interoperability that match the risks these systems pose.
We argue that criteria for these choices may be drawn from emerging subfields
within antitrust, tort, and administrative law. It will then be possible for
courts, federal and state agencies, and non-governmental organizations to play
more active roles in RL specification and evaluation. Building on the "model
cards" and "datasheets" frameworks proposed by Mitchell et al. and Gebru et
al., we argue the need for Reward Reports for AI systems. Reward Reports are
living documents for proposed RL deployments that demarcate design choices.
