Parameter Sharing For Heterogeneous Agents in Multi-Agent Reinforcement
  Learning
  Parameter sharing, where each agent independently learns a policy with fully
shared parameters between all policies, is a popular baseline method for
multi-agent deep reinforcement learning. Unfortunately, since all agents share
the same policy network, they cannot learn different policies or tasks. This
issue has been circumvented experimentally by adding an agent-specific
indicator signal to observations, which we term "agent indication." Agent
indication is limited, however, in that without modification it does not allow
parameter sharing to be applied to environments where the action spaces and/or
observation spaces are heterogeneous. This work formalizes the notion of agent
indication and proves that it enables convergence to optimal policies for the
first time. Next, we formally introduce methods to extend parameter sharing to
learning in heterogeneous observation and action spaces, and prove that these
methods allow for convergence to optimal policies. Finally, we experimentally
confirm that the methods we introduce function empirically, and conduct a wide
array of experiments studying the empirical efficacy of many different agent
indication schemes for graphical observation spaces.
