Cooperative and Competitive Biases for Multi-Agent Reinforcement
  Learning
  Training a multi-agent reinforcement learning (MARL) algorithm is more
challenging than training a single-agent reinforcement learning algorithm,
because the result of a multi-agent task strongly depends on the complex
interactions among agents and their interactions with a stochastic and dynamic
environment. We propose an algorithm that boosts MARL training using the biased
action information of other agents based on a friend-or-foe concept. For a
cooperative and competitive environment, there are generally two groups of
agents: cooperative-agents and competitive-agents. In the proposed algorithm,
each agent updates its value function using its own action and the biased
action information of other agents in the two groups. The biased joint action
of cooperative agents is computed as the sum of their actual joint action and
the imaginary cooperative joint action, by assuming all the cooperative agents
jointly maximize the target agent's value function. The biased joint action of
competitive agents can be computed similarly. Each agent then updates its own
value function using the biased action information, resulting in a biased value
function and corresponding biased policy. Subsequently, the biased policy of
each agent is inevitably subjected to recommend an action to cooperate and
compete with other agents, thereby introducing more active interactions among
agents and enhancing the MARL policy learning. We empirically demonstrate that
our algorithm outperforms existing algorithms in various mixed
cooperative-competitive environments. Furthermore, the introduced biases
gradually decrease as the training proceeds and the correction based on the
imaginary assumption vanishes.
