Large Associative Memory Problem in Neurobiology and Machine Learning
  Dense Associative Memories or modern Hopfield networks permit storage and
reliable retrieval of an exponentially large (in the dimension of feature
space) number of memories. At the same time, their naive implementation is
non-biological, since it seemingly requires the existence of many-body synaptic
junctions between the neurons. We show that these models are effective
descriptions of a more microscopic (written in terms of biological degrees of
freedom) theory that has additional (hidden) neurons and only requires two-body
interactions between them. For this reason our proposed microscopic theory is a
valid model of large associative memory with a degree of biological
plausibility. The dynamics of our network and its reduced dimensional
equivalent both minimize energy (Lyapunov) functions. When certain dynamical
variables (hidden neurons) are integrated out from our microscopic theory, one
can recover many of the models that were previously discussed in the
literature, e.g. the model presented in "Hopfield Networks is All You Need"
paper. We also provide an alternative derivation of the energy function and the
update rule proposed in the aforementioned paper and clarify the relationships
between various models of this class.
