Hierarchical Deep Double Q-Routing
  This paper explores a deep reinforcement learning approach applied to the
packet routing problem with high-dimensional constraints instigated by dynamic
and autonomous communication networks. Our approach is motivated by the fact
that centralized path calculation approaches are often not scalable, whereas
the distributed approaches with locally acting nodes are not fully aware of the
end-to-end performance. We instead hierarchically distribute the path
calculation over designated nodes in the network while taking into account the
end-to-end performance. Specifically, we develop a hierarchical
cluster-oriented adaptive per-flow path calculation mechanism by leveraging the
Deep Double Q-network (DDQN) algorithm, where the end-to-end paths are
calculated by the source nodes with the assistance of cluster (group) leaders
at different hierarchical levels. In our approach, a deferred composite reward
is designed to capture the end-to-end performance through a feedback signal
from the source nodes to the group leaders and captures the local network
performance through the local resource assessments by the group leaders. This
approach scales in large networks, adapts to the dynamic demand, utilizes the
network resources efficiently and can be applied to segment routing.
