Do ideas have shape? Plato's theory of forms as the continuous limit of
  artificial neural networks
  We show that ResNets converge, in the infinite depth limit, to a
generalization of image registration algorithms. In this generalization, images
are replaced by abstractions (ideas) living in high dimensional RKHS spaces,
and material points are replaced by data points. Whereas computational anatomy
aligns images via deformations of the material space, this generalization
aligns ideas by via transformations of their RKHS. This identification of
ResNets as idea registration algorithms has several remarkable consequences.
The search for good architectures can be reduced to that of good kernels, and
we show that the composition of idea registration blocks with reduced
equivariant multi-channel kernels (introduced here) recovers and generalizes
CNNs to arbitrary spaces and groups of transformations. Minimizers of $L_2$
regularized ResNets satisfy a discrete least action principle implying the near
preservation of the norm of weights and biases across layers. The parameters of
trained ResNets can be identified as solutions of an autonomous Hamiltonian
system defined by the activation function and the architecture of the ANN.
Momenta variables provide a sparse representation of the parameters of a
ResNet. The registration regularization strategy provides a provably robust
alternative to Dropout for ANNs. Pointwise RKHS error estimates lead to
deterministic error estimates for ANNs.
