DiffCloud: Real-to-Sim from Point Clouds with Differentiable Simulation
  and Rendering of Deformable Objects
  Research in manipulation of deformable objects is typically conducted on a
limited range of scenarios, because handling each scenario on hardware takes
significant effort. Realistic simulators with support for various types of
deformations and interactions have the potential to speed up experimentation
with novel tasks and algorithms. However, for highly deformable objects it is
challenging to align the output of a simulator with the behavior of real
objects. Manual tuning is not intuitive, hence automated methods are needed. We
view this alignment problem as a joint perception-inference challenge and
demonstrate how to use recent neural network architectures to successfully
perform simulation parameter inference from real point clouds. We analyze the
performance of various architectures, comparing their data and training
requirements. Furthermore, we propose to leverage differentiable point cloud
sampling and differentiable simulation to significantly reduce the time to
achieve the alignment. We employ an efficient way to propagate gradients from
point clouds to simulated meshes and further through to the physical simulation
parameters, such as mass and stiffness. Experiments with highly deformable
objects show that our method can achieve comparable or better alignment with
real object behavior, while reducing the time needed to achieve this by more
than an order of magnitude. Videos and supplementary material are available at
https://tinyurl.com/diffcloud.
