Reinforcement Learning with Function-Valued Action Spaces for Partial
  Differential Equation Control
  Recent work has shown that reinforcement learning (RL) is a promising
approach to control dynamical systems described by partial differential
equations (PDE). This paper shows how to use RL to tackle more general PDE
control problems that have continuous high-dimensional action spaces with
spatial relationship among action dimensions. In particular, we propose the
concept of action descriptors, which encode regularities among
spatially-extended action dimensions and enable the agent to control
high-dimensional action PDEs. We provide theoretical evidence suggesting that
this approach can be more sample efficient compared to a conventional approach
that treats each action dimension separately and does not explicitly exploit
the spatial regularity of the action space. The action descriptor approach is
then used within the deep deterministic policy gradient algorithm. Experiments
on two PDE control problems, with up to 256-dimensional continuous actions,
show the advantage of the proposed approach over the conventional one.
