title: GAN diverges after 3rd warm restart
body: <p>I've been working with a ClusterGAN, originally based on the one here: <a href="https://github.com/zhampel/clusterGAN" rel="nofollow noreferrer">https://github.com/zhampel/clusterGAN</a></p>
  <p>While trying to stabilize training on my data I integrated a bunch of current strategies, like making it fully convolutional, including spectral norm, using warm restarts, and adding instance noise. Though I was able to solve problems I was having with mode collapse, and I can get useful output images, it will eventually diverge and I'm wondering what might be going on (see chart below). To be clear, the output of the model is actually quite usable around epoch 69 (in this run), right before the last restart, but I'm still curious what might be wrong, partly for my own understanding, but also to inform future exploration. Is this overfitting, perhaps?</p>
  <p>Worth noting, perhaps, is that a larger version of the model—just using more filters—did tend to diverge <em>later</em> in the training, but would still diverge eventually (and didn't necessarily produce clearly &quot;better&quot; images). I'm running on mobile, so I'd like to make the model as small as possible, both to improve inference speed and keep app size down.</p>
  <p><a href="https://i.stack.imgur.com/NOtM9.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/NOtM9.png" alt="enter image description here" /></a></p>
  <p>UPDATE: Increasing the initial learning rate stabilized training for this number of epochs (though I'm curious whether it would still diverge if I ran it through another warm restart cycle).</p>
  <p><a href="https://i.stack.imgur.com/cxreu.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/cxreu.png" alt="enter image description here" /></a></p>