Learning Multi-Object Dynamics with Compositional Neural Radiance Fields
  We present a method to learn compositional predictive models from image
observations based on implicit object encoders, Neural Radiance Fields (NeRFs),
and graph neural networks. A central question in learning dynamic models from
sensor observations is on which representations predictions should be
performed. NeRFs have become a popular choice for representing scenes due to
their strong 3D prior. However, most NeRF approaches are trained on a single
scene, representing the whole scene with a global model, making generalization
to novel scenes, containing different numbers of objects, challenging. Instead,
we present a compositional, object-centric auto-encoder framework that maps
multiple views of the scene to a \emph{set} of latent vectors representing each
object separately. The latent vectors parameterize individual NeRF models from
which the scene can be reconstructed and rendered from novel viewpoints. We
train a graph neural network dynamics model in the latent space to achieve
compositionality for dynamics prediction. A key feature of our approach is that
the learned 3D information of the scene through the NeRF model enables us to
incorporate structural priors in learning the dynamics models, making long-term
predictions more stable. The model can further be used to synthesize new scenes
from individual object observations. For planning, we utilize RRTs in the
learned latent space, where we can exploit our model and the implicit object
encoder to make sampling the latent space informative and more efficient. In
the experiments, we show that the model outperforms several baselines on a
pushing task containing many objects. Video:
https://dannydriess.github.io/compnerfdyn/
