Learning Latent Graph Dynamics for Visual Manipulation of Deformable
  Objects
  Manipulating deformable objects, such as ropes and clothing, is a
long-standing challenge in robotics, because of their large degrees of freedom,
complex non-linear dynamics, and self-occlusion in visual perception. The key
difficulty is a suitable representation, rich enough to capture the object
shape, dynamics for manipulation and yet simple enough to be estimated reliably
from visual observations. This work aims to learn latent Graph dynamics for
DefOrmable Object Manipulation (G-DOOM). G-DOOM approximates a deformable
object as a sparse set of interacting keypoints, which are extracted
automatically from images via unsupervised learning. It learns a graph neural
network that captures abstractly the geometry and the interaction dynamics of
the keypoints. To handle object self-occlusion, G-DOOM uses a recurrent neural
network to track the keypoints over time and condition their interactions on
the history. We then train the resulting recurrent graph dynamics model through
contrastive learning in a high-fidelity simulator. For manipulation planning,
G-DOOM reasons explicitly about the learned dynamics model through
model-predictive control applied at each keypoint. Preliminary experiments of
G-DOOM on a set of challenging rope and cloth manipulation tasks indicate
strong performance, compared with state-of-the-art methods. Although trained in
a simulator, G-DOOM transfers directly to a real robot for both rope and cloth
manipulation.
