Leveraging Communication Topologies Between Learning Agents in Deep
  Reinforcement Learning
  A common technique to improve learning performance in deep reinforcement
learning (DRL) and many other machine learning algorithms is to run multiple
learning agents in parallel. A neglected component in the development of these
algorithms has been how best to arrange the learning agents involved to improve
distributed search. Here we draw upon results from the networked optimization
literatures suggesting that arranging learning agents in communication networks
other than fully connected topologies (the implicit way agents are commonly
arranged in) can improve learning. We explore the relative performance of four
popular families of graphs and observe that one such family (Erdos-Renyi random
graphs) empirically outperforms the de facto fully-connected communication
topology across several DRL benchmark tasks. Additionally, we observe that 1000
learning agents arranged in an Erdos-Renyi graph can perform as well as 3000
agents arranged in the standard fully-connected topology, showing the large
learning improvement possible when carefully designing the topology over which
agents communicate. We complement these empirical results with a theoretical
investigation of why our alternate topologies perform better. Overall, our work
suggests that distributed machine learning algorithms could be made more
effective if the communication topology between learning agents was optimized.
