Decomposition Methods with Deep Corrections for Reinforcement Learning
  Decomposition methods have been proposed to approximate solutions to large
sequential decision making problems. In contexts where an agent interacts with
multiple entities, utility decomposition can be used to separate the global
objective into local tasks considering each individual entity independently. An
arbitrator is then responsible for combining the individual utilities and
selecting an action in real time to solve the global problem. Although these
techniques can perform well empirically, they rely on strong assumptions of
independence between the local tasks and sacrifice the optimality of the global
solution. This paper proposes an approach that improves upon such approximate
solutions by learning a correction term represented by a neural network. We
demonstrate this approach on a fisheries management problem where multiple
boats must coordinate to maximize their catch over time as well as on a
pedestrian avoidance problem for autonomous driving. In each problem,
decomposition methods can scale to multiple boats or pedestrians by using
strategies involving one entity. We verify empirically that the proposed
correction method significantly improves the decomposition method and
outperforms a policy trained on the full scale problem without utility
decomposition.
