Gradient Layer: Enhancing the Convergence of Adversarial Training for
  Generative Models
  We propose a new technique that boosts the convergence of training generative
adversarial networks. Generally, the rate of training deep models reduces
severely after multiple iterations. A key reason for this phenomenon is that a
deep network is expressed using a highly non-convex finite-dimensional model,
and thus the parameter gets stuck in a local optimum. Because of this, methods
often suffer not only from degeneration of the convergence speed but also from
limitations in the representational power of the trained network. To overcome
this issue, we propose an additional layer called the gradient layer to seek a
descent direction in an infinite-dimensional space. Because the layer is
constructed in the infinite-dimensional space, we are not restricted by the
specific model structure of finite-dimensional models. As a result, we can get
out of the local optima in finite-dimensional models and move towards the
global optimal function more directly. In this paper, this phenomenon is
explained from the functional gradient method perspective of the gradient
layer. Interestingly, the optimization procedure using the gradient layer
naturally constructs the deep structure of the network. Moreover, we
demonstrate that this procedure can be regarded as a discretization method of
the gradient flow that naturally reduces the objective function. Finally, the
method is tested using several numerical experiments, which show its fast
convergence.
