Learning Collaborative Policies to Solve NP-hard Routing Problems
  Recently, deep reinforcement learning (DRL) frameworks have shown potential
for solving NP-hard routing problems such as the traveling salesman problem
(TSP) without problem-specific expert knowledge. Although DRL can be used to
solve complex problems, DRL frameworks still struggle to compete with
state-of-the-art heuristics showing a substantial performance gap. This paper
proposes a novel hierarchical problem-solving strategy, termed learning
collaborative policies (LCP), which can effectively find the near-optimum
solution using two iterative DRL policies: the seeder and reviser. The seeder
generates as diversified candidate solutions as possible (seeds) while being
dedicated to exploring over the full combinatorial action space (i.e., sequence
of assignment action). To this end, we train the seeder's policy using a simple
yet effective entropy regularization reward to encourage the seeder to find
diverse solutions. On the other hand, the reviser modifies each candidate
solution generated by the seeder; it partitions the full trajectory into
sub-tours and simultaneously revises each sub-tour to minimize its traveling
distance. Thus, the reviser is trained to improve the candidate solution's
quality, focusing on the reduced solution space (which is beneficial for
exploitation). Extensive experiments demonstrate that the proposed two-policies
collaboration scheme improves over single-policy DRL framework on various
NP-hard routing problems, including TSP, prize collecting TSP (PCTSP), and
capacitated vehicle routing problem (CVRP).
