PreCNet: Next Frame Video Prediction Based on Predictive Coding
  Predictive coding, currently a highly influential theory in neuroscience, has
not been widely adopted in machine learning yet. In this work, we transform the
seminal model of Rao and Ballard (1999) into a modern deep learning framework
while remaining maximally faithful to the original schema. The resulting
network we propose (PreCNet) is tested on a widely used next frame video
prediction benchmark, which consists of images from an urban environment
recorded from a car-mounted camera. On this benchmark (training: 41k images
from KITTI dataset; testing: Caltech Pedestrian dataset), we achieve to our
knowledge the best performance to date when measured with the Structural
Similarity Index (SSIM). Performance on all measures was further improved when
a larger training set (2M images from BDD100k), pointing to the limitations of
the KITTI training set. This work demonstrates that an architecture carefully
based in a neuroscience model, without being explicitly tailored to the task at
hand, can exhibit unprecedented performance.
