Dynamics-Aware Comparison of Learned Reward Functions
  The ability to learn reward functions plays an important role in enabling the
deployment of intelligent agents in the real world. However, comparing reward
functions, for example as a means of evaluating reward learning methods,
presents a challenge. Reward functions are typically compared by considering
the behavior of optimized policies, but this approach conflates deficiencies in
the reward function with those of the policy search algorithm used to optimize
it. To address this challenge, Gleave et al. (2020) propose the
Equivalent-Policy Invariant Comparison (EPIC) distance. EPIC avoids policy
optimization, but in doing so requires computing reward values at transitions
that may be impossible under the system dynamics. This is problematic for
learned reward functions because it entails evaluating them outside of their
training distribution, resulting in inaccurate reward values that we show can
render EPIC ineffective at comparing rewards. To address this problem, we
propose the Dynamics-Aware Reward Distance (DARD), a new reward pseudometric.
DARD uses an approximate transition model of the environment to transform
reward functions into a form that allows for comparisons that are invariant to
reward shaping while only evaluating reward functions on transitions close to
their training distribution. Experiments in simulated physical domains
demonstrate that DARD enables reliable reward comparisons without policy
optimization and is significantly more predictive than baseline methods of
downstream policy performance when dealing with learned reward functions.
