Generative Planning for Temporally Coordinated Exploration in
  Reinforcement Learning
  Standard model-free reinforcement learning algorithms optimize a policy that
generates the action to be taken in the current time step in order to maximize
expected future return. While flexible, it faces difficulties arising from the
inefficient exploration due to its single step nature. In this work, we present
Generative Planning method (GPM), which can generate actions not only for the
current step, but also for a number of future steps (thus termed as generative
planning). This brings several benefits to GPM. Firstly, since GPM is trained
by maximizing value, the plans generated from it can be regarded as intentional
action sequences for reaching high value regions. GPM can therefore leverage
its generated multi-step plans for temporally coordinated exploration towards
high value regions, which is potentially more effective than a sequence of
actions generated by perturbing each action at single step level, whose
consistent movement decays exponentially with the number of exploration steps.
Secondly, starting from a crude initial plan generator, GPM can refine it to be
adaptive to the task, which, in return, benefits future explorations. This is
potentially more effective than commonly used action-repeat strategy, which is
non-adaptive in its form of plans. Additionally, since the multi-step plan can
be interpreted as the intent of the agent from now to a span of time period
into the future, it offers a more informative and intuitive signal for
interpretation. Experiments are conducted on several benchmark environments and
the results demonstrated its effectiveness compared with several baseline
methods.
