Inverse Reinforcement Learning via Nonparametric Spatio-Temporal Subgoal
  Modeling
  Advances in the field of inverse reinforcement learning (IRL) have led to
sophisticated inference frameworks that relax the original modeling assumption
of observing an agent behavior that reflects only a single intention. Instead
of learning a global behavioral model, recent IRL methods divide the
demonstration data into parts, to account for the fact that different
trajectories may correspond to different intentions, e.g., because they were
generated by different domain experts. In this work, we go one step further:
using the intuitive concept of subgoals, we build upon the premise that even a
single trajectory can be explained more efficiently locally within a certain
context than globally, enabling a more compact representation of the observed
behavior. Based on this assumption, we build an implicit intentional model of
the agent's goals to forecast its behavior in unobserved situations. The result
is an integrated Bayesian prediction framework that significantly outperforms
existing IRL solutions and provides smooth policy estimates consistent with the
expert's plan. Most notably, our framework naturally handles situations where
the intentions of the agent change over time and classical IRL algorithms fail.
In addition, due to its probabilistic nature, the model can be
straightforwardly applied in active learning scenarios to guide the
demonstration process of the expert.
