Prediction, Consistency, Curvature: Representation Learning for
  Locally-Linear Control
  Many real-world sequential decision-making problems can be formulated as
optimal control with high-dimensional observations and unknown dynamics. A
promising approach is to embed the high-dimensional observations into a
lower-dimensional latent representation space, estimate the latent dynamics
model, then utilize this model for control in the latent space. An important
open question is how to learn a representation that is amenable to existing
control algorithms? In this paper, we focus on learning representations for
locally-linear control algorithms, such as iterative LQR (iLQR). By formulating
and analyzing the representation learning problem from an optimal control
perspective, we establish three underlying principles that the learned
representation should comprise: 1) accurate prediction in the observation
space, 2) consistency between latent and observation space dynamics, and 3) low
curvature in the latent space transitions. These principles naturally
correspond to a loss function that consists of three terms: prediction,
consistency, and curvature (PCC). Crucially, to make PCC tractable, we derive
an amortized variational bound for the PCC loss function. Extensive experiments
on benchmark domains demonstrate that the new variational-PCC learning
algorithm benefits from significantly more stable and reproducible training,
and leads to superior control performance. Further ablation studies give
support to the importance of all three PCC components for learning a good
latent space for control.
