Your Policy Regularizer is Secretly an Adversary
  Policy regularization methods such as maximum entropy regularization are
widely used in reinforcement learning to improve the robustness of a learned
policy. In this paper, we show how this robustness arises from hedging against
worst-case perturbations of the reward function, which are chosen from a
limited set by an imagined adversary. Using convex duality, we characterize
this robust set of adversarial reward perturbations under KL and
alpha-divergence regularization, which includes Shannon and Tsallis entropy
regularization as special cases. Importantly, generalization guarantees can be
given within this robust set. We provide detailed discussion of the worst-case
reward perturbations, and present intuitive empirical examples to illustrate
this robustness and its relationship with generalization. Finally, we discuss
how our analysis complements and extends previous results on adversarial reward
robustness and path consistency optimality conditions.
