Continual 3D Convolutional Neural Networks for Real-time Processing of
  Videos
  This paper introduces Continual 3D Convolutional Neural Networks (Co3D CNNs),
a new computational formulation of spatio-temporal 3D CNNs, in which videos are
processed frame-by-frame rather than by clip. In online processing tasks
demanding frame-wise predictions, Co3D CNNs dispense with the computational
redundancies of regular 3D CNNs, namely the repeated convolutions over frames,
which appear in overlapping clips. We show that Continual 3D CNNs can reuse
preexisting 3D-CNN weights to reduce the per-prediction floating point
operations (FLOPs) in proportion to the temporal receptive field while
retaining similar memory requirements and accuracy. This is validated with
multiple models on the Kinetics-400 and Charades datasets with remarkable
results: Continual X3D models attain state-of-the-art complexity/accuracy
trade-offs on Kinetics-400 with 12.1-15.3x reductions of FLOPs and 2.3-3.8%
improvements in accuracy compared to regular X3D models while reducing peak
memory consumption by up to 48%. Moreover, we investigate the transient
response of Co3D CNNs at start-up and perform an extensive benchmark of
on-hardware processing speed and accuracy for publicly available 3D CNNs.
