SoftHebb: Bayesian inference in unsupervised Hebbian soft
  winner-take-all networks
  State-of-the-art artificial neural networks (ANNs) require labelled data or
feedback between layers, are often biologically implausible, and are vulnerable
to adversarial attacks that humans are not susceptible to. On the other hand,
Hebbian learning in winner-take-all (WTA) networks, is unsupervised,
feed-forward, and biologically plausible. However, a modern objective
optimization theory for WTA networks has been missing, except under very
limiting assumptions. Here we derive formally such a theory, based on
biologically plausible but generic ANN elements. Through Hebbian learning,
network parameters maintain a Bayesian generative model of the data. There is
no supervisory loss function, but the network does minimize cross-entropy
between its activations and the input distribution. The key is a "soft" WTA
where there is no absolute "hard" winner neuron, and a specific type of
Hebbian-like plasticity of weights and biases. We confirm our theory in
practice, where, in handwritten digit (MNIST) recognition, our Hebbian
algorithm, SoftHebb, minimizes cross-entropy without having access to it, and
outperforms the more frequently used, hard-WTA-based method. Strikingly, it
even outperforms supervised end-to-end backpropagation, under certain
conditions. Specifically, in a two-layered network, SoftHebb outperforms
backpropagation when the training dataset is only presented once, when the
testing data is noisy, and under gradient-based adversarial attacks. Notably,
adversarial attacks that confuse SoftHebb are also confusing to the human eye.
Finally, the model can generate interpolations of objects from its input
distribution. All in all, SoftHebb extends Hebbian WTA theory with modern
machine learning tools, thus making these networks relevant to pertinent issues
in deep learning.
