How to Fill the Optimum Set? Population Gradient Descent with Harmless
  Diversity
  Although traditional optimization methods focus on finding a single optimal
solution, most objective functions in modern machine learning problems,
especially those in deep learning, often have multiple or infinite numbers of
optima. Therefore, it is useful to consider the problem of finding a set of
diverse points in the optimum set of an objective function. In this work, we
frame this problem as a bi-level optimization problem of maximizing a diversity
score inside the optimum set of the main loss function, and solve it with a
simple population gradient descent framework that iteratively updates the
points to maximize the diversity score in a fashion that does not hurt the
optimization of the main loss. We demonstrate that our method can efficiently
generate diverse solutions on a variety of applications, including
text-to-image generation, text-to-mesh generation, molecular conformation
generation and ensemble neural network training.
