Variational Characterizations of Local Entropy and Heat Regularization
  in Deep Learning
  The aim of this paper is to provide new theoretical and computational
understanding on two loss regularizations employed in deep learning, known as
local entropy and heat regularization. For both regularized losses we introduce
variational characterizations that naturally suggest a two-step scheme for
their optimization, based on the iterative shift of a probability density and
the calculation of a best Gaussian approximation in Kullback-Leibler
divergence. Under this unified light, the optimization schemes for local
entropy and heat regularized loss differ only over which argument of the
Kullback-Leibler divergence is used to find the best Gaussian approximation.
Local entropy corresponds to minimizing over the second argument, and the
solution is given by moment matching. This allows to replace traditional
back-propagation calculation of gradients by sampling algorithms, opening an
avenue for gradient-free, parallelizable training of neural networks.
