Optimization in Machine Learning: A Distribution Space Approach
  We present the viewpoint that optimization problems encountered in machine
learning can often be interpreted as minimizing a convex functional over a
function space, but with a non-convex constraint set introduced by model
parameterization. This observation allows us to repose such problems via a
suitable relaxation as convex optimization problems in the space of
distributions over the training parameters. We derive some simple relationships
between the distribution-space problem and the original problem, e.g. a
distribution-space solution is at least as good as a solution in the original
space. Moreover, we develop a numerical algorithm based on mixture
distributions to perform approximate optimization directly in distribution
space. Consistency of this approximation is established and the numerical
efficacy of the proposed algorithm is illustrated on simple examples. In both
theory and practice, this formulation provides an alternative approach to
large-scale optimization in machine learning.
