Feudal Multi-Agent Reinforcement Learning with Adaptive Network
  Partition for Traffic Signal Control
  Multi-agent reinforcement learning (MARL) has been applied and shown great
potential in multi-intersections traffic signal control, where multiple agents,
one for each intersection, must cooperate together to optimize traffic flow. To
encourage global cooperation, previous work partitions the traffic network into
several regions and learns policies for agents in a feudal structure. However,
static network partition fails to adapt to dynamic traffic flow, which will
changes frequently over time. To address this, we propose a novel feudal MARL
approach with adaptive network partition. Specifically, we first partition the
network into several regions according to the traffic flow. To do this, we
propose two approaches: one is directly to use graph neural network (GNN) to
generate the network partition, and the other is to use Monte-Carlo tree search
(MCTS) to find the best partition with criteria computed by GNN. Then, we
design a variant of Qmix using GNN to handle various dimensions of input, given
by the dynamic network partition. Finally, we use a feudal hierarchy to manage
agents in each partition and promote global cooperation. By doing so, agents
are able to adapt to the traffic flow as required in practice. We empirically
evaluate our method both in a synthetic traffic grid and real-world traffic
networks of three cities, widely used in the literature. Our experimental
results confirm that our method can achieve better performance, in terms of
average travel time and queue length, than several leading methods for traffic
signal control.
