Associative Memory in Iterated Overparameterized Sigmoid Autoencoders
  Recent work showed that overparameterized autoencoders can be trained to
implement associative memory via iterative maps, when the trained input-output
Jacobian of the network has all of its eigenvalue norms strictly below one.
Here, we theoretically analyze this phenomenon for sigmoid networks by
leveraging recent developments in deep learning theory, especially the
correspondence between training neural networks in the infinite-width limit and
performing kernel regression with the Neural Tangent Kernel (NTK). We find that
overparameterized sigmoid autoencoders can have attractors in the NTK limit for
both training with a single example and multiple examples under certain
conditions. In particular, for multiple training examples, we find that the
norm of the largest Jacobian eigenvalue drops below one with increasing input
norm, leading to associative memory.
