Joint Policy Search for Multi-agent Collaboration with Imperfect
  Information
  To learn good joint policies for multi-agent collaboration with imperfect
information remains a fundamental challenge. While for two-player zero-sum
games, coordinate-ascent approaches (optimizing one agent's policy at a time,
e.g., self-play) work with guarantees, in multi-agent cooperative setting they
often converge to sub-optimal Nash equilibrium. On the other hand, directly
modeling joint policy changes in imperfect information game is nontrivial due
to complicated interplay of policies (e.g., upstream updates affect downstream
state reachability). In this paper, we show global changes of game values can
be decomposed to policy changes localized at each information set, with a novel
term named policy-change density. Based on this, we propose Joint Policy
Search(JPS) that iteratively improves joint policies of collaborative agents in
imperfect information games, without re-evaluating the entire game. On
multi-agent collaborative tabular games, JPS is proven to never worsen
performance and can improve solutions provided by unilateral approaches (e.g,
CFR), outperforming algorithms designed for collaborative policy learning (e.g.
BAD). Furthermore, for real-world games, JPS has an online form that naturally
links with gradient updates. We test it to Contract Bridge, a 4-player
imperfect-information game where a team of $2$ collaborates to compete against
the other. In its bidding phase, players bid in turn to find a good contract
through a limited information channel. Based on a strong baseline agent that
bids competitive bridge purely through domain-agnostic self-play, JPS improves
collaboration of team players and outperforms WBridge5, a championship-winning
software, by $+0.63$ IMPs (International Matching Points) per board over 1k
games, substantially better than previous SoTA ($+0.41$ IMPs/b) under
Double-Dummy evaluation.
