title: Keras LSTM - Categorical Cross Entropy falls to 0
body: <p>I'm currently trying to compare some RNNs and I have an issue only with the LSTM and I have no idea why. </p>
  
  <p>I'm training with the same code/dataset a LSTM, SimpleRNN and GRU. For all of them the loss decrease normally. But for the LSTM, after a certain point (loss around 0.4), the loss directly falls to 10e-8. If I try to predict an output, I only have Nan.</p>
  
  <p>This is the code :</p>
  
  <pre><code>nb_unit = 7
  inp_shape = (maxlen, 7)
  loss_ = "categorical_crossentropy"
  metrics_ = "categorical_crossentropy"
  optimizer_ = "Nadam"
  nb_epoch = 250
  batch_size = 64
  
  model = Sequential()
  
  model.add(LSTM( units=nb_unit, 
                  input_shape=inp_shape, 
                  return_sequences=True, 
                  activation='softmax'))  # I just change the cell name
  model.compile(loss=loss_,
                optimizer=optimizer_,
                metrics=[metrics_])
  
  checkpoint = ModelCheckpoint("lstm_simple.h5",
                              monitor=loss_,
                              verbose=1,
                              save_best_only=True,
                              save_weights_only=False,
                              mode='auto',
                              period=1)
  early = EarlyStopping( monitor='loss',
                         min_delta=0,
                         patience=10,
                         verbose=1,
                         mode='auto')
  
  history = model.fit(X_train, y_train, 
                      validation_data=(X_test, y_test), 
                      epochs=nb_epoch, 
                      batch_size=batch_size, 
                      verbose=2, 
                      callbacks = [checkpoint, early])
  </code></pre>
  
  <p>This the the output of GRU and LSTM with the same input :</p>
  
  <pre><code>Input :
  [[[1 0 0 0 0 0 0]
    [0 1 0 0 0 0 0]
    [0 0 0 1 0 0 0]
    [0 0 0 1 0 0 0]
    [0 1 0 0 0 0 0]
    [0 0 0 0 0 1 0]
    [0 0 0 0 1 0 0]
    [0 0 0 1 0 0 0]
    [0 0 0 0 0 1 0]
    [0 0 0 0 1 0 0]
    [0 0 0 1 0 0 0]
    [0 1 0 0 0 0 0]
    [0 0 0 0 0 1 0]
    [0 0 0 0 1 0 0]
    [0 0 0 1 0 0 0]
    [0 0 0 0 0 1 0]
    [0 0 0 0 0 1 0]
    [0 0 0 0 0 0 0]
    [0 0 0 0 0 0 0]
    [0 0 0 0 0 0 0]]]
  
  
  LSTM predicts :
  [[[ nan  nan  nan  nan  nan  nan  nan]
    [ nan  nan  nan  nan  nan  nan  nan]
    [ nan  nan  nan  nan  nan  nan  nan]
    [ nan  nan  nan  nan  nan  nan  nan]
    [ nan  nan  nan  nan  nan  nan  nan]
    [ nan  nan  nan  nan  nan  nan  nan]
    [ nan  nan  nan  nan  nan  nan  nan]
    [ nan  nan  nan  nan  nan  nan  nan]
    [ nan  nan  nan  nan  nan  nan  nan]
    [ nan  nan  nan  nan  nan  nan  nan]
    [ nan  nan  nan  nan  nan  nan  nan]
    [ nan  nan  nan  nan  nan  nan  nan]
    [ nan  nan  nan  nan  nan  nan  nan]
    [ nan  nan  nan  nan  nan  nan  nan]
    [ nan  nan  nan  nan  nan  nan  nan]
    [ nan  nan  nan  nan  nan  nan  nan]
    [ nan  nan  nan  nan  nan  nan  nan]
    [ nan  nan  nan  nan  nan  nan  nan]
    [ nan  nan  nan  nan  nan  nan  nan]
    [ nan  nan  nan  nan  nan  nan  nan]]]
  
  
  GRU predicts :
  [[[ 0.     0.54   0.     0.     0.407  0.     0.   ]
    [ 0.     0.005  0.66   0.314  0.     0.     0.001]
    [ 0.     0.001  0.032  0.957  0.     0.004  0.   ]
    [ 0.     0.628  0.     0.     0.     0.372  0.   ]
    [ 0.     0.555  0.     0.     0.     0.372  0.   ]
    [ 0.     0.     0.     0.     0.996  0.319  0.   ]
    [ 0.     0.     0.167  0.55   0.     0.     0.   ]
    [ 0.     0.486  0.     0.002  0.     0.51   0.   ]
    [ 0.     0.001  0.     0.     0.992  0.499  0.   ]
    [ 0.     0.     0.301  0.55   0.     0.     0.   ]
    [ 0.     0.396  0.001  0.007  0.     0.592  0.   ]
    [ 0.     0.689  0.     0.     0.     0.592  0.   ]
    [ 0.     0.001  0.     0.     0.997  0.592  0.   ]
    [ 0.     0.     0.37   0.55   0.     0.     0.   ]
    [ 0.     0.327  0.003  0.025  0.     0.599  0.   ]
    [ 0.     0.001  0.     0.     0.967  0.599  0.002]
    [ 0.     0.     0.     0.     0.     0.002  0.874]
    [ 0.004  0.076  0.128  0.337  0.02   0.069  0.378]
    [ 0.006  0.379  0.047  0.113  0.029  0.284  0.193]
    [ 0.006  0.469  0.001  0.037  0.13   0.295  0.193]]]
  </code></pre>
  
  <p>For the loss, you can find below the last lines of the fit() history:</p>
  
  <pre><code>Epoch 116/250
  Epoch 00116: categorical_crossentropy did not improve
   - 2s - loss: 0.3774 - categorical_crossentropy: 0.3774 - val_loss: 0.3945 - val_categorical_crossentropy: 0.3945
  
  Epoch 117/250
  Epoch 00117: categorical_crossentropy improved from 0.37673 to 0.08198, saving model to lstm_simple.h5
   - 2s - loss: 0.0820 - categorical_crossentropy: 0.0820 - val_loss: 7.8743e-08 - val_categorical_crossentropy: 7.8743e-08
  
  Epoch 118/250
  Epoch 00118: categorical_crossentropy improved from 0.08198 to 0.00000, saving model to lstm_simple.h5
   - 2s - loss: 7.5460e-08 - categorical_crossentropy: 7.5460e-08 - val_loss: 7.8743e-08 - val_categorical_crossentropy: 7.8743e-08
  </code></pre>
  
  <p>Or the evolution of the loss based on Epochs.</p>
  
  <p><a href="https://i.stack.imgur.com/G9Yui.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/G9Yui.png" alt="enter image description here"></a></p>
  
  <p>I previously tried it without Softmax and with MSE as loss function and I didn't get any error.</p>
  
  <p>If needed, you can find the notebook and script to generate the dataset on Github (<a href="https://github.com/Coni63/SO/blob/master/Reber.ipynb" rel="nofollow noreferrer">https://github.com/Coni63/SO/blob/master/Reber.ipynb</a>).</p>
  
  <p>Many thanks for your support,
  Regards,
  Nicolas</p>
  
  <h3>EDIT 1:</h3>
  
  <p>The root cause seems to be the Softmax function which vanished. If I stop it before it crashed and display the sum of the softmax for every timestep I have :</p>
  
  <pre><code>LSTM :
  [[ 0.112]
   [ 0.008]
   [ 0.379]
   [ 0.04 ]
   [ 0.001]
   [ 0.104]
   [ 0.021]
   [ 0.   ]
   [ 0.104]
   [ 0.343]
   [ 0.012]
   [ 0.   ]
   [ 0.23 ]
   [ 0.13 ]
   [ 0.147]
   [ 0.145]
   [ 0.152]
   [ 0.157]
   [ 0.163]
   [ 0.169]]
  
  
  GRU :
  [[ 0.974]
   [ 0.807]
   [ 0.719]
   [ 1.184]
   [ 0.944]
   [ 0.999]
   [ 1.426]
   [ 0.957]
   [ 0.999]
   [ 1.212]
   [ 1.52 ]
   [ 0.954]
   [ 0.42 ]
   [ 0.83 ]
   [ 0.903]
   [ 0.944]
   [ 0.976]
   [ 1.005]
   [ 1.022]
   [ 1.029]]
  </code></pre>
  
  <p>With a Softmax of 0, the next step will try to divide by 0. Now I have no idea how to fix it.</p>