Selective Dyna-style Planning Under Limited Model Capacity
  In model-based reinforcement learning, planning with an imperfect model of
the environment has the potential to harm learning progress. But even when a
model is imperfect, it may still contain information that is useful for
planning. In this paper, we investigate the idea of using an imperfect model
selectively. The agent should plan in parts of the state space where the model
would be helpful but refrain from using the model where it would be harmful. An
effective selective planning mechanism requires estimating predictive
uncertainty, which arises out of aleatoric uncertainty, parameter uncertainty,
and model inadequacy, among other sources. Prior work has focused on parameter
uncertainty for selective planning. In this work, we emphasize the importance
of model inadequacy. We show that heteroscedastic regression can signal
predictive uncertainty arising from model inadequacy that is complementary to
that which is detected by methods designed for parameter uncertainty,
indicating that considering both parameter uncertainty and model inadequacy may
be a more promising direction for effective selective planning than either in
isolation.
