On Solving Cooperative MARL Problems with a Few Good Experiences
  Cooperative Multi-agent Reinforcement Learning (MARL) is crucial for
cooperative decentralized decision learning in many domains such as search and
rescue, drone surveillance, package delivery and fire fighting problems. In
these domains, a key challenge is learning with a few good experiences, i.e.,
positive reinforcements are obtained only in a few situations (e.g., on
extinguishing a fire or tracking a crime or delivering a package) and in most
other situations there is zero or negative reinforcement. Learning decisions
with a few good experiences is extremely challenging in cooperative MARL
problems due to three reasons. First, compared to the single agent case,
exploration is harder as multiple agents have to be coordinated to receive a
good experience. Second, environment is not stationary as all the agents are
learning at the same time (and hence change policies). Third, scale of problem
increases significantly with every additional agent.
  Relevant existing work is extensive and has focussed on dealing with a few
good experiences in single-agent RL problems or on scalable approaches for
handling non-stationarity in MARL problems. Unfortunately, neither of these
approaches (or their extensions) are able to address the problem of sparse good
experiences effectively. Therefore, we provide a novel fictitious self
imitation approach that is able to simultaneously handle non-stationarity and
sparse good experiences in a scalable manner. Finally, we provide a thorough
comparison (experimental or descriptive) against relevant cooperative MARL
algorithms to demonstrate the utility of our approach.
