R2-B2: Recursive Reasoning-Based Bayesian Optimization for No-Regret
  Learning in Games
  This paper presents a recursive reasoning formalism of Bayesian optimization
(BO) to model the reasoning process in the interactions between boundedly
rational, self-interested agents with unknown, complex, and costly-to-evaluate
payoff functions in repeated games, which we call Recursive Reasoning-Based BO
(R2-B2). Our R2-B2 algorithm is general in that it does not constrain the
relationship among the payoff functions of different agents and can thus be
applied to various types of games such as constant-sum, general-sum, and
common-payoff games. We prove that by reasoning at level 2 or more and at one
level higher than the other agents, our R2-B2 agent can achieve faster
asymptotic convergence to no regret than that without utilizing recursive
reasoning. We also propose a computationally cheaper variant of R2-B2 called
R2-B2-Lite at the expense of a weaker convergence guarantee. The performance
and generality of our R2-B2 algorithm are empirically demonstrated using
synthetic games, adversarial machine learning, and multi-agent reinforcement
learning.
