Dictionary Learning by Dynamical Neural Networks
  A dynamical neural network consists of a set of interconnected neurons that
interact over time continuously. It can exhibit computational properties in the
sense that the dynamical system's evolution and/or limit points in the
associated state space can correspond to numerical solutions to certain
mathematical optimization or learning problems. Such a computational system is
particularly attractive in that it can be mapped to a massively parallel
computer architecture for power and throughput efficiency, especially if each
neuron can rely solely on local information (i.e., local memory). Deriving
gradients from the dynamical network's various states while conforming to this
last constraint, however, is challenging. We show that by combining ideas of
top-down feedback and contrastive learning, a dynamical network for solving the
l1-minimizing dictionary learning problem can be constructed, and the true
gradients for learning are provably computable by individual neurons. Using
spiking neurons to construct our dynamical network, we present a learning
process, its rigorous mathematical analysis, and numerical results on several
dictionary learning problems.
