MBB: Model-Based Baseline for Global Guidance of Model-Free
  Reinforcement Learning via Lower-Dimensional Solutions
  One spectrum on which robotic control paradigms lie is the degree in which a
model of the environment is involved, from methods that are completely
model-free such as model-free RL, to methods that require a known model such as
optimal control, with other methods such as model-based RL somewhere in the
middle. On one end of the spectrum, model-free RL can learn control policies
for high-dimensional (hi-dim), complex robotic tasks through trial-and-error
without knowledge of a model of the environment, but tends to require a large
amount of data. On the other end, "classical methods" such as optimal control
generate solutions without collecting data, but assume that an accurate model
of the system and environment is known and are mostly limited to problems with
low-dimensional (lo-dim) state spaces. In this paper, we bring the two ends of
the spectrum together. Although models of hi-dim systems and environments may
not exist, lo-dim approximations of these systems and environments are widely
available, especially in robotics. Therefore, we propose to solve hi-dim,
complex robotic tasks in two stages. First, assuming a coarse model of the
hi-dim system, we compute a lo-dim value function for the lo-dim version of the
problem using classical methods (eg. value iteration and optimal control).
Then, the lo-dim value function is used as a baseline function to warm-start
the model-free RL process that learns hi-dim policies. The lo-dim value
function provides global guidance for model-free RL, alleviating the data
inefficiency of model-free RL. We demonstrate our approach on two robot
learning tasks with hi-dim state spaces and observe significant improvement in
policy performance and learning efficiency. We also give an empirical analysis
of our method with a third task.
