How Do Neural Networks Estimate Optical Flow? A Neuropsychology-Inspired
  Study
  End-to-end trained convolutional neural networks have led to a breakthrough
in optical flow estimation. The most recent advances focus on improving the
optical flow estimation by improving the architecture and setting a new
benchmark on the publicly available MPI-Sintel dataset. Instead, in this
article, we investigate how deep neural networks estimate optical flow. A
better understanding of how these networks function is important for (i)
assessing their generalization capabilities to unseen inputs, and (ii)
suggesting changes to improve their performance. For our investigation, we
focus on FlowNetS, as it is the prototype of an encoder-decoder neural network
for optical flow estimation. Furthermore, we use a filter identification method
that has played a major role in uncovering the motion filters present in animal
brains in neuropsychological research. The method shows that the filters in the
deepest layer of FlowNetS are sensitive to a variety of motion patterns. Not
only do we find translation filters, as demonstrated in animal brains, but
thanks to the easier measurements in artificial neural networks, we even unveil
dilation, rotation, and occlusion filters. Furthermore, we find similarities in
the refinement part of the network and the perceptual filling-in process which
occurs in the mammal primary visual cortex.
