Learning Compact Models for Planning with Exogenous Processes
  We address the problem of approximate model minimization for MDPs in which
the state is partitioned into endogenous and (much larger) exogenous
components. An exogenous state variable is one whose dynamics are independent
of the agent's actions. We formalize the mask-learning problem, in which the
agent must choose a subset of exogenous state variables to reason about when
planning; doing planning in such a reduced state space can often be
significantly more efficient than planning in the full model. We then explore
the various value functions at play within this setting, and describe
conditions under which a policy for a reduced model will be optimal for the
full MDP. The analysis leads us to a tractable approximate algorithm that draws
upon the notion of mutual information among exogenous state variables. We
validate our approach in simulated robotic manipulation domains where a robot
is placed in a busy environment, in which there are many other agents also
interacting with the objects. Visit http://tinyurl.com/chitnis-exogenous for a
supplementary video.
