SIMONe: View-Invariant, Temporally-Abstracted Object Representations via
  Unsupervised Video Decomposition
  To help agents reason about scenes in terms of their building blocks, we wish
to extract the compositional structure of any given scene (in particular, the
configuration and characteristics of objects comprising the scene). This
problem is especially difficult when scene structure needs to be inferred while
also estimating the agent's location/viewpoint, as the two variables jointly
give rise to the agent's observations. We present an unsupervised variational
approach to this problem. Leveraging the shared structure that exists across
different scenes, our model learns to infer two sets of latent representations
from RGB video input alone: a set of "object" latents, corresponding to the
time-invariant, object-level contents of the scene, as well as a set of "frame"
latents, corresponding to global time-varying elements such as viewpoint. This
factorization of latents allows our model, SIMONe, to represent object
attributes in an allocentric manner which does not depend on viewpoint.
Moreover, it allows us to disentangle object dynamics and summarize their
trajectories as time-abstracted, view-invariant, per-object properties. We
demonstrate these capabilities, as well as the model's performance in terms of
view synthesis and instance segmentation, across three procedurally generated
video datasets.
