A Deep Reinforcement Learning Architecture for Multi-stage Optimal
  Control
  Deep reinforcement learning for high dimensional, hierarchical control tasks
usually requires the use of complex neural networks as functional
approximators, which can lead to inefficiency, instability and even divergence
in the training process. Here, we introduce stacked deep Q learning (SDQL), a
flexible modularized deep reinforcement learning architecture, that can enable
finding of optimal control policy of control tasks consisting of multiple
linear stages in a stable and efficient way. SDQL exploits the linear stage
structure by approximating the Q function via a collection of deep Q
sub-networks stacking along an axis marking the stage-wise progress of the
whole task. By back-propagating the learned state values from later stages to
earlier stages, all sub-networks co-adapt to maximize the total reward of the
whole task, although each sub-network is responsible for learning optimal
control policy for its own stage. This modularized architecture offers
considerable flexibility in terms of environment and policy modeling, as it
allows choices of different state spaces, action spaces, reward structures, and
Q networks for each stage, Further, the backward stage-wise training procedure
of SDQL can offers additional transparency, stability, and flexibility to the
training process, thus facilitating model fine-tuning and hyper-parameter
search. We demonstrate that SDQL is capable of learning competitive strategies
for problems with characteristics of high-dimensional state space,
heterogeneous action space(both discrete and continuous), multiple scales, and
sparse and delayed rewards.
