A Solvable High-Dimensional Model of GAN
  We present a theoretical analysis of the training process for a single-layer
GAN fed by high-dimensional input data. The training dynamics of the proposed
model at both microscopic and macroscopic scales can be exactly analyzed in the
high-dimensional limit. In particular, we prove that the macroscopic quantities
measuring the quality of the training process converge to a deterministic
process characterized by an ordinary differential equation (ODE), whereas the
microscopic states containing all the detailed weights remain stochastic, whose
dynamics can be described by a stochastic differential equation (SDE). This
analysis provides a new perspective different from recent analyses in the limit
of small learning rate, where the microscopic state is always considered
deterministic, and the contribution of noise is ignored. From our analysis, we
show that the level of the background noise is essential to the convergence of
the training process: setting the noise level too strong leads to failure of
feature recovery, whereas setting the noise too weak causes oscillation.
Although this work focuses on a simple copy model of GAN, we believe the
analysis methods and insights developed here would prove useful in the
theoretical understanding of other variants of GANs with more advanced training
algorithms.
