Unsupervised Learning of Lagrangian Dynamics from Images for Prediction
  and Control
  Recent approaches for modelling dynamics of physical systems with neural
networks enforce Lagrangian or Hamiltonian structure to improve prediction and
generalization. However, when coordinates are embedded in high-dimensional data
such as images, these approaches either lose interpretability or can only be
applied to one particular example. We introduce a new unsupervised neural
network model that learns Lagrangian dynamics from images, with
interpretability that benefits prediction and control. The model infers
Lagrangian dynamics on generalized coordinates that are simultaneously learned
with a coordinate-aware variational autoencoder (VAE). The VAE is designed to
account for the geometry of physical systems composed of multiple rigid bodies
in the plane. By inferring interpretable Lagrangian dynamics, the model learns
physical system properties, such as kinetic and potential energy, which enables
long-term prediction of dynamics in the image space and synthesis of
energy-based controllers.
