Stable Distribution Alignment Using the Dual of the Adversarial Distance
  Methods that align distributions by minimizing an adversarial distance
between them have recently achieved impressive results. However, these
approaches are difficult to optimize with gradient descent and they often do
not converge well without careful hyperparameter tuning and proper
initialization. We investigate whether turning the adversarial min-max problem
into an optimization problem by replacing the maximization part with its dual
improves the quality of the resulting alignment and explore its connections to
Maximum Mean Discrepancy. Our empirical results suggest that using the dual
formulation for the restricted family of linear discriminators results in a
more stable convergence to a desirable solution when compared with the
performance of a primal min-max GAN-like objective and an MMD objective under
the same restrictions. We test our hypothesis on the problem of aligning two
synthetic point clouds on a plane and on a real-image domain adaptation problem
on digits. In both cases, the dual formulation yields an iterative procedure
that gives more stable and monotonic improvement over time.
