On Learning Intrinsic Rewards for Policy Gradient Methods
  In many sequential decision making tasks, it is challenging to design reward
functions that help an RL agent efficiently learn behavior that is considered
good by the agent designer. A number of different formulations of the
reward-design problem, or close variants thereof, have been proposed in the
literature. In this paper we build on the Optimal Rewards Framework of Singh
et.al. that defines the optimal intrinsic reward function as one that when used
by an RL agent achieves behavior that optimizes the task-specifying or
extrinsic reward function. Previous work in this framework has shown how good
intrinsic reward functions can be learned for lookahead search based planning
agents. Whether it is possible to learn intrinsic reward functions for learning
agents remains an open problem. In this paper we derive a novel algorithm for
learning intrinsic rewards for policy-gradient based learning agents. We
compare the performance of an augmented agent that uses our algorithm to
provide additive intrinsic rewards to an A2C-based policy learner (for Atari
games) and a PPO-based policy learner (for Mujoco domains) with a baseline
agent that uses the same policy learners but with only extrinsic rewards. Our
results show improved performance on most but not all of the domains.
