Unsupervised Video Decomposition using Spatio-temporal Iterative
  Inference
  Unsupervised multi-object scene decomposition is a fast-emerging problem in
representation learning. Despite significant progress in static scenes, such
models are unable to leverage important dynamic cues present in video. We
propose a novel spatio-temporal iterative inference framework that is powerful
enough to jointly model complex multi-object representations and explicit
temporal dependencies between latent variables across frames. This is achieved
by leveraging 2D-LSTM, temporally conditioned inference and generation within
the iterative amortized inference for posterior refinement. Our method improves
the overall quality of decompositions, encodes information about the objects'
dynamics, and can be used to predict trajectories of each object separately.
Additionally, we show that our model has a high accuracy even without color
information. We demonstrate the decomposition, segmentation, and prediction
capabilities of our model and show that it outperforms the state-of-the-art on
several benchmark datasets, one of which was curated for this work and will be
made publicly available.
