Improving Generalization of Reinforcement Learning with Minimax
  Distributional Soft Actor-Critic
  Reinforcement learning (RL) has achieved remarkable performance in numerous
sequential decision making and control tasks. However, a common problem is that
learned nearly optimal policy always overfits to the training environment and
may not be extended to situations never encountered during training. For
practical applications, the randomness of environment usually leads to some
devastating events, which should be the focus of safety-critical systems such
as autonomous driving. In this paper, we introduce the minimax formulation and
distributional framework to improve the generalization ability of RL algorithms
and develop the Minimax Distributional Soft Actor-Critic (Minimax DSAC)
algorithm. Minimax formulation aims to seek optimal policy considering the most
severe variations from environment, in which the protagonist policy maximizes
action-value function while the adversary policy tries to minimize it.
Distributional framework aims to learn a state-action return distribution, from
which we can model the risk of different returns explicitly, thereby
formulating a risk-averse protagonist policy and a risk-seeking adversarial
policy. We implement our method on the decision-making tasks of autonomous
vehicles at intersections and test the trained policy in distinct environments.
Results demonstrate that our method can greatly improve the generalization
ability of the protagonist agent to different environmental variations.
