DeltaCNN: End-to-End CNN Inference of Sparse Frame Differences in Videos
  Convolutional neural network inference on video data requires powerful
hardware for real-time processing. Given the inherent coherence across
consecutive frames, large parts of a video typically change little. By skipping
identical image regions and truncating insignificant pixel updates,
computational redundancy can in theory be reduced significantly. However, these
theoretical savings have been difficult to translate into practice, as sparse
updates hamper computational consistency and memory access coherence; which are
key for efficiency on real hardware. With DeltaCNN, we present a sparse
convolutional neural network framework that enables sparse frame-by-frame
updates to accelerate video inference in practice. We provide sparse
implementations for all typical CNN layers and propagate sparse feature updates
end-to-end - without accumulating errors over time. DeltaCNN is applicable to
all convolutional neural networks without retraining. To the best of our
knowledge, we are the first to significantly outperform the dense reference,
cuDNN, in practical settings, achieving speedups of up to 7x with only marginal
differences in accuracy.
