ExPoSe: Combining State-Based Exploration with Gradient-Based Online
  Search
  A tree-based online search algorithm iteratively simulates trajectories and
updates Q-value information on a set of states represented by a tree structure.
Alternatively, policy gradient based online search algorithms update the
information obtained from simulated trajectories directly onto the parameters
of the policy and has been found to be effective. While tree-based methods
limit the updates from simulations to the states that exist in the tree and do
not interpolate the information to nearby states, policy gradient search
methods do not do explicit exploration. In this paper, we show that it is
possible to combine and leverage the strengths of these two methods for
improved search performance. We examine the key reasons behind the improvement
and propose a simple yet effective online search method, named Exploratory
Policy Gradient Search (ExPoSe), that updates both the parameters of the policy
as well as search information on the states in the trajectory. We conduct
experiments on complex planning problems, which include Sokoban and Hamiltonian
cycle search in sparse graphs and show that combining exploration with policy
gradient improves online search performance.
