Neural Non-Rigid Tracking
  We introduce a novel, end-to-end learnable, differentiable non-rigid tracker
that enables state-of-the-art non-rigid reconstruction by a learned robust
optimization. Given two input RGB-D frames of a non-rigidly moving object, we
employ a convolutional neural network to predict dense correspondences and
their confidences. These correspondences are used as constraints in an
as-rigid-as-possible (ARAP) optimization problem. By enabling gradient
back-propagation through the weighted non-linear least squares solver, we are
able to learn correspondences and confidences in an end-to-end manner such that
they are optimal for the task of non-rigid tracking. Under this formulation,
correspondence confidences can be learned via self-supervision, informing a
learned robust optimization, where outliers and wrong correspondences are
automatically down-weighted to enable effective tracking. Compared to
state-of-the-art approaches, our algorithm shows improved reconstruction
performance, while simultaneously achieving 85 times faster correspondence
prediction than comparable deep-learning based methods. We make our code
available.
