DeepCQ+: Robust and Scalable Routing with Multi-Agent Deep Reinforcement
  Learning for Highly Dynamic Networks
  Highly dynamic mobile ad-hoc networks (MANETs) remain as one of the most
challenging environments to develop and deploy robust, efficient, and scalable
routing protocols. In this paper, we present DeepCQ+ routing protocol which, in
a novel manner integrates emerging multi-agent deep reinforcement learning
(MADRL) techniques into existing Q-learning-based routing protocols and their
variants and achieves persistently higher performance across a wide range of
topology and mobility configurations. While keeping the overall protocol
structure of the Q-learning-based routing protocols, DeepCQ+ replaces
statically configured parameterized thresholds and hand-written rules with
carefully designed MADRL agents such that no configuration of such parameters
is required a priori. Extensive simulation shows that DeepCQ+ yields
significantly increased end-to-end throughput with lower overhead and no
apparent degradation of end-to-end delays (hop counts) compared to its
Q-learning based counterparts. Qualitatively, and perhaps more significantly,
DeepCQ+ maintains remarkably similar performance gains under many scenarios
that it was not trained for in terms of network sizes, mobility conditions, and
traffic dynamics. To the best of our knowledge, this is the first successful
application of the MADRL framework for the MANET routing problem that
demonstrates a high degree of scalability and robustness even under
environments that are outside the trained range of scenarios. This implies that
our MARL-based DeepCQ+ design solution significantly improves the performance
of Q-learning based CQ+ baseline approach for comparison and increases its
practicality and explainability because the real-world MANET environment will
likely vary outside the trained range of MANET scenarios. Additional techniques
to further increase the gains in performance and scalability are discussed.
