Continuously Discovering Novel Strategies via Reward-Switching Policy
  Optimization
  We present Reward-Switching Policy Optimization (RSPO), a paradigm to
discover diverse strategies in complex RL environments by iteratively finding
novel policies that are both locally optimal and sufficiently different from
existing ones. To encourage the learning policy to consistently converge
towards a previously undiscovered local optimum, RSPO switches between
extrinsic and intrinsic rewards via a trajectory-based novelty measurement
during the optimization process. When a sampled trajectory is sufficiently
distinct, RSPO performs standard policy optimization with extrinsic rewards.
For trajectories with high likelihood under existing policies, RSPO utilizes an
intrinsic diversity reward to promote exploration. Experiments show that RSPO
is able to discover a wide spectrum of strategies in a variety of domains,
ranging from single-agent particle-world tasks and MuJoCo continuous control to
multi-agent stag-hunt games and StarCraftII challenges.
