GraN-GAN: Piecewise Gradient Normalization for Generative Adversarial
  Networks
  Modern generative adversarial networks (GANs) predominantly use piecewise
linear activation functions in discriminators (or critics), including ReLU and
LeakyReLU. Such models learn piecewise linear mappings, where each piece
handles a subset of the input space, and the gradients per subset are piecewise
constant. Under such a class of discriminator (or critic) functions, we present
Gradient Normalization (GraN), a novel input-dependent normalization method,
which guarantees a piecewise K-Lipschitz constraint in the input space. In
contrast to spectral normalization, GraN does not constrain processing at the
individual network layers, and, unlike gradient penalties, strictly enforces a
piecewise Lipschitz constraint almost everywhere. Empirically, we demonstrate
improved image generation performance across multiple datasets (incl.
CIFAR-10/100, STL-10, LSUN bedrooms, and CelebA), GAN loss functions, and
metrics. Further, we analyze altering the often untuned Lipschitz constant K in
several standard GANs, not only attaining significant performance gains, but
also finding connections between K and training dynamics, particularly in
low-gradient loss plateaus, with the common Adam optimizer.
