Assisted Robust Reward Design
  Real-world robotic tasks require complex reward functions. When we define the
problem the robot needs to solve, we pretend that a designer specifies this
complex reward exactly, and it is set in stone from then on. In practice,
however, reward design is an iterative process: the designer chooses a reward,
eventually encounters an "edge-case" environment where the reward incentivizes
the wrong behavior, revises the reward, and repeats. What would it mean to
rethink robotics problems to formally account for this iterative nature of
reward design? We propose that the robot not take the specified reward for
granted, but rather have uncertainty about it, and account for the future
design iterations as future evidence. We contribute an Assisted Reward Design
method that speeds up the design process by anticipating and influencing this
future evidence: rather than letting the designer eventually encounter failure
cases and revise the reward then, the method actively exposes the designer to
such environments during the development phase. We test this method in a
simplified autonomous driving task and find that it more quickly improves the
car's behavior in held-out environments by proposing environments that are
"edge cases" for the current reward.
