Dyna-T: Dyna-Q and Upper Confidence Bounds Applied to Trees
  In this work we present a preliminary investigation of a novel algorithm
called Dyna-T. In reinforcement learning (RL) a planning agent has its own
representation of the environment as a model. To discover an optimal policy to
interact with the environment, the agent collects experience in a trial and
error fashion. Experience can be used for learning a better model or improve
directly the value function and policy. Typically separated, Dyna-Q is an
hybrid approach which, at each iteration, exploits the real experience to
update the model as well as the value function, while planning its action using
simulated data from its model. However, the planning process is computationally
expensive and strongly depends on the dimensionality of the state-action space.
We propose to build a Upper Confidence Tree (UCT) on the simulated experience
and search for the best action to be selected during the on-line learning
process. We prove the effectiveness of our proposed method on a set of
preliminary tests on three testbed environments from Open AI. In contrast to
Dyna-Q, Dyna-T outperforms state-of-the-art RL agents in the stochastic
environments by choosing a more robust action selection strategy.
