Modeling Strong and Human-Like Gameplay with KL-Regularized Search
  We consider the task of building strong but human-like policies in
multi-agent decision-making problems, given examples of human behavior.
Imitation learning is effective at predicting human actions but may not match
the strength of expert humans, while self-play learning and search techniques
(e.g. AlphaZero) lead to strong performance but may produce policies that are
difficult for humans to understand and coordinate with. We show in chess and Go
that regularizing search based on the KL divergence from an imitation-learned
policy results in higher human prediction accuracy and stronger performance
than imitation learning alone. We then introduce a novel regret minimization
algorithm that is regularized based on the KL divergence from an
imitation-learned policy, and show that using this algorithm for search in
no-press Diplomacy yields a policy that matches the human prediction accuracy
of imitation learning while being substantially stronger.
