Greedy Hierarchical Variational Autoencoders for Large-Scale Video
  Prediction
  A video prediction model that generalizes to diverse scenes would enable
intelligent agents such as robots to perform a variety of tasks via planning
with the model. However, while existing video prediction models have produced
promising results on small datasets, they suffer from severe underfitting when
trained on large and diverse datasets. To address this underfitting challenge,
we first observe that the ability to train larger video prediction models is
often bottlenecked by the memory constraints of GPUs or TPUs. In parallel, deep
hierarchical latent variable models can produce higher quality predictions by
capturing the multi-level stochasticity of future observations, but end-to-end
optimization of such models is notably difficult. Our key insight is that
greedy and modular optimization of hierarchical autoencoders can simultaneously
address both the memory constraints and the optimization challenges of
large-scale video prediction. We introduce Greedy Hierarchical Variational
Autoencoders (GHVAEs), a method that learns high-fidelity video predictions by
greedily training each level of a hierarchical autoencoder. In comparison to
state-of-the-art models, GHVAEs provide 17-55% gains in prediction performance
on four video datasets, a 35-40% higher success rate on real robot tasks, and
can improve performance monotonically by simply adding more modules.
