Normative Disagreement as a Challenge for Cooperative AI
  Cooperation in settings where agents have both common and conflicting
interests (mixed-motive environments) has recently received considerable
attention in multi-agent learning. However, the mixed-motive environments
typically studied have a single cooperative outcome on which all agents can
agree. Many real-world multi-agent environments are instead bargaining problems
(BPs): they have several Pareto-optimal payoff profiles over which agents have
conflicting preferences. We argue that typical cooperation-inducing learning
algorithms fail to cooperate in BPs when there is room for normative
disagreement resulting in the existence of multiple competing cooperative
equilibria, and illustrate this problem empirically. To remedy the issue, we
introduce the notion of norm-adaptive policies. Norm-adaptive policies are
capable of behaving according to different norms in different circumstances,
creating opportunities for resolving normative disagreement. We develop a class
of norm-adaptive policies and show in experiments that these significantly
increase cooperation. However, norm-adaptiveness cannot address residual
bargaining failure arising from a fundamental tradeoff between exploitability
and cooperative robustness.
