PlaNet of the Bayesians: Reconsidering and Improving Deep Planning
  Network by Incorporating Bayesian Inference
  In the present paper, we propose an extension of the Deep Planning Network
(PlaNet), also referred to as PlaNet of the Bayesians (PlaNet-Bayes). There has
been a growing demand in model predictive control (MPC) in partially observable
environments in which complete information is unavailable because of, for
example, lack of expensive sensors. PlaNet is a promising solution to realize
such latent MPC, as it is used to train state-space models via model-based
reinforcement learning (MBRL) and to conduct planning in the latent space.
However, recent state-of-the-art strategies mentioned in MBRR literature, such
as involving uncertainty into training and planning, have not been considered,
significantly suppressing the training performance. The proposed extension is
to make PlaNet uncertainty-aware on the basis of Bayesian inference, in which
both model and action uncertainty are incorporated. Uncertainty in latent
models is represented using a neural network ensemble to approximately infer
model posteriors. The ensemble of optimal action candidates is also employed to
capture multimodal uncertainty in the optimality. The concept of the action
ensemble relies on a general variational inference MPC (VI-MPC) framework and
its instance, probabilistic action ensemble with trajectory sampling (PaETS).
In this paper, we extend VI-MPC and PaETS, which have been originally
introduced in previous literature, to address partially observable cases. We
experimentally compare the performances on continuous control tasks, and
conclude that our method can consistently improve the asymptotic performance
compared with PlaNet.
