Dual Training of Energy-Based Models with Overparametrized Shallow
  Neural Networks
  Energy-based models (EBMs) are generative models that are usually trained via
maximum likelihood estimation. This approach becomes challenging in generic
situations where the trained energy is non-convex, due to the need to sample
the Gibbs distribution associated with this energy. Using general Fenchel
duality results, we derive variational principles dual to maximum likelihood
EBMs with shallow overparametrized neural network energies, both in the
feature-learning and lazy linearized regimes. In the feature-learning regime,
this dual formulation justifies using a two time-scale gradient ascent-descent
(GDA) training algorithm in which one updates concurrently the particles in the
sample space and the neurons in the parameter space of the energy. We also
consider a variant of this algorithm in which the particles are sometimes
restarted at random samples drawn from the data set, and show that performing
these restarts at every iteration step corresponds to score matching training.
These results are illustrated in simple numerical experiments, which indicates
that GDA performs best when features and particles are updated using similar
time scales.
