Using Videos to Evaluate Image Model Robustness
  Human visual systems are robust to a wide range of image transformations that
are challenging for artificial networks. We present the first study of image
model robustness to the minute transformations found across video frames, which
we term "natural robustness". Compared to previous studies on adversarial
examples and synthetic distortions, natural robustness captures a more diverse
set of common image transformations that occur in the natural environment. Our
study across a dozen model architectures shows that more accurate models are
more robust to natural transformations, and that robustness to synthetic color
distortions is a good proxy for natural robustness. In examining brittleness in
videos, we find that majority of the brittleness found in videos lies outside
the typical definition of adversarial examples (99.9\%). Finally, we
investigate training techniques to reduce brittleness and find that no single
technique systematically improves natural robustness across twelve tested
architectures.
