title: Training loss increases after 12 epochs
body: <p>I have a model that learns to classify (binary classification) almost at 100% accuracy after 7-14 epochs but after reaching the minimum loss of 0.0004, in the next epoch it jumps to as much as 7.5 (which means it has a 50% chance of classifying correctly, same has pure chance) and then stays near 7. for all subsequent epochs.</p>
  
  <p>I use adam optimiser which should take care of the learning rate.</p>
  
  <p>How can I prevent the training loss from increasing? </p>
  
  <p>This huge jump doesn't happen for SGD optimiser.</p>
  
  
  
  <pre class="lang-python prettyprint-override"><code>inputs = Input(shape=(X_train.shape[1],))
  Dx = Dense(32, activation="relu")(inputs)
  Dx = Dense(32, activation="relu")(Dx)
  for i in range(20):
      Dx = Dense(32, activation="relu")(Dx)
  Dx = Dense(1, activation="sigmoid")(Dx)
  D = Model(input=[inputs], output=[Dx])
  D.compile(loss="binary_crossentropy", optimizer="adam")
  
  D.fit(X_train, y_train, nb_epoch=20)
  </code></pre>