Rethinking Generative Mode Coverage: A Pointwise Guaranteed Approach
  Many generative models have to combat $\textit{missing modes}$. The
conventional wisdom to this end is by reducing through training a statistical
distance (such as $f$-divergence) between the generated distribution and
provided data distribution. But this is more of a heuristic than a guarantee.
The statistical distance measures a $\textit{global}$, but not
$\textit{local}$, similarity between two distributions. Even if it is small, it
does not imply a plausible mode coverage. Rethinking this problem from a
game-theoretic perspective, we show that a complete mode coverage is firmly
attainable. If a generative model can approximate a data distribution
moderately well under a global statistical distance measure, then we will be
able to find a mixture of generators that collectively covers $\textit{every}$
data point and thus $\textit{every}$ mode, with a lower-bounded generation
probability. Constructing the generator mixture has a connection to the
multiplicative weights update rule, upon which we propose our algorithm. We
prove that our algorithm guarantees complete mode coverage. And our experiments
on real and synthetic datasets confirm better mode coverage over recent
approaches, ones that also use generator mixtures but rely on global
statistical distances.
