Optimizing Variational Representations of Divergences and Accelerating
  their Statistical Estimation
  Variational representations of divergences and distances between
high-dimensional probability distributions offer significant theoretical
insights and practical advantages in numerous research areas. Recently, they
have gained popularity in machine learning as a tractable and scalable approach
for training probabilistic models and for statistically differentiating between
data distributions. Their advantages include: 1) They can be estimated from
data as statistical averages. 2) Such representations can leverage the ability
of neural networks to efficiently approximate optimal solutions in function
spaces. However, a systematic and practical approach to improving the tightness
of such variational formulas, and accordingly accelerate statistical learning
and estimation from data, is currently lacking. Here we develop such a
methodology for building new, tighter variational representations of
divergences. Our approach relies on improved objective functionals constructed
via an auxiliary optimization problem. Furthermore, the calculation of the
functional Hessian of objective functionals unveils the local curvature
differences around the common optimal variational solution; this quantifies and
orders the tightness gains between different variational representations.
Finally, numerical simulations utilizing neural network optimization
demonstrate that tighter representations can result in significantly faster
learning and more accurate estimation of divergences in both synthetic and real
datasets (of more than 1000 dimensions), often accelerated by nearly an order
of magnitude.
