Simplifying Reward Design through Divide-and-Conquer
  Designing a good reward function is essential to robot planning and
reinforcement learning, but it can also be challenging and frustrating. The
reward needs to work across multiple different environments, and that often
requires many iterations of tuning. We introduce a novel divide-and-conquer
approach that enables the designer to specify a reward separately for each
environment. By treating these separate reward functions as observations about
the underlying true reward, we derive an approach to infer a common reward
across all environments. We conduct user studies in an abstract grid world
domain and in a motion planning domain for a 7-DOF manipulator that measure
user effort and solution quality. We show that our method is faster, easier to
use, and produces a higher quality solution than the typical method of
designing a reward jointly across all environments. We additionally conduct a
series of experiments that measure the sensitivity of these results to
different properties of the reward design task, such as the number of
environments, the number of feasible solutions per environment, and the
fraction of the total features that vary within each environment. We find that
independent reward design outperforms the standard, joint, reward design
process but works best when the design problem can be divided into simpler
subproblems.
