Can convolutional ResNets approximately preserve input distances? A
  frequency analysis perspective
  ResNets constrained to be bi-Lipschitz, that is, approximately distance
preserving, have been a crucial component of recently proposed techniques for
deterministic uncertainty quantification in neural models. We show that
theoretical justifications for recent regularisation schemes trying to enforce
such a constraint suffer from a crucial flaw -- the theoretical link between
the regularisation scheme used and bi-Lipschitzness is only valid under
conditions which do not hold in practice, rendering existing theory of limited
use, despite the strong empirical performance of these models. We provide a
theoretical explanation for the effectiveness of these regularisation schemes
using a frequency analysis perspective, showing that under mild conditions
these schemes will enforce a lower Lipschitz bound on the low-frequency
projection of images. We then provide empirical evidence supporting our
theoretical claims, and perform further experiments which demonstrate that our
broader conclusions appear to hold when some of the mathematical assumptions of
our proof are relaxed, corresponding to the setup used in prior work. In
addition, we present a simple constructive algorithm to search for counter
examples to the distance preservation condition, and discuss possible
implications of our theory for future model design.
