Learning Belief Representations for Imitation Learning in POMDPs
  We consider the problem of imitation learning from expert demonstrations in
partially observable Markov decision processes (POMDPs). Belief
representations, which characterize the distribution over the latent states in
a POMDP, have been modeled using recurrent neural networks and probabilistic
latent variable models, and shown to be effective for reinforcement learning in
POMDPs. In this work, we investigate the belief representation learning problem
for generative adversarial imitation learning in POMDPs. Instead of training
the belief module and the policy separately as suggested in prior work, we
learn the belief module jointly with the policy, using a task-aware imitation
loss to ensure that the representation is more aligned with the policy's
objective. To improve robustness of representation, we introduce several
informative belief regularization techniques, including multi-step prediction
of dynamics and action-sequences. Evaluated on various partially observable
continuous-control locomotion tasks, our belief-module imitation learning
approach (BMIL) substantially outperforms several baselines, including the
original GAIL algorithm and the task-agnostic belief learning algorithm.
Extensive ablation analysis indicates the effectiveness of task-aware belief
learning and belief regularization.
