Scalable, Decentralized Multi-Agent Reinforcement Learning Methods
  Inspired by Stigmergy and Ant Colonies
  Bolstering multi-agent learning algorithms to tackle complex coordination and
control tasks has been a long-standing challenge of on-going research. Numerous
methods have been proposed to help reduce the effects of non-stationarity and
unscalability. In this work, we investigate a novel approach to decentralized
multi-agent learning and planning that attempts to address these two
challenges. In particular, this method is inspired by the cohesion,
coordination, and behavior of ant colonies. As a result, these algorithms are
designed to be naturally scalable to systems with numerous agents. While no
optimality is guaranteed, the method is intended to work well in practice and
scale better in efficacy with the number of agents present than others. The
approach combines single-agent RL and an ant-colony-inspired decentralized,
stigmergic algorithm for multi-agent path planning and environment
modification. Specifically, we apply this algorithm in a setting where agents
must navigate to a goal location, learning to push rectangular boxes into holes
to yield new traversable pathways. It is shown that while the approach yields
promising success in this particular environment, it may not be as easily
generalized to others. The algorithm designed is notably scalable to numerous
agents but is limited in its performance due to its relatively simplistic,
rule-based approach. Furthermore, the composability of RL-trained policies is
called into question, where, while policies are successful in their training
environments, applying trained policies to a larger-scale, multi-agent
framework results in unpredictable behavior.
