title: Keras training crashes mid epoch after multiple correct executions
body: <p>I am trying create a Cudgru based model that predicts sequence of 7 features that are interrelated. Here's my keras model summary:</p>
  
  <pre><code>_________________________________________________________________
  Layer (type)                 Output Shape              Param #
  =================================================================
  cu_dnngru_1 (CuDNNGRU)       (None, 49, 100)           32700
  _________________________________________________________________
  dropout_1 (Dropout)          (None, 49, 100)           0
  _________________________________________________________________
  cu_dnngru_2 (CuDNNGRU)       (None, 49, 100)           60600
  _________________________________________________________________
  dropout_2 (Dropout)          (None, 49, 100)           0
  _________________________________________________________________
  cu_dnngru_3 (CuDNNGRU)       (None, 49, 100)           60600
  _________________________________________________________________
  dropout_3 (Dropout)          (None, 49, 100)           0
  _________________________________________________________________
  cu_dnngru_4 (CuDNNGRU)       (None, 49, 100)           60600
  _________________________________________________________________
  dropout_4 (Dropout)          (None, 49, 100)           0
  _________________________________________________________________
  cu_dnngru_5 (CuDNNGRU)       (None, 49, 100)           60600
  _________________________________________________________________
  dropout_5 (Dropout)          (None, 49, 100)           0
  _________________________________________________________________
  cu_dnngru_6 (CuDNNGRU)       (None, 49, 100)           60600
  _________________________________________________________________
  dropout_6 (Dropout)          (None, 49, 100)           0
  _________________________________________________________________
  cu_dnngru_7 (CuDNNGRU)       (None, 49, 100)           60600
  _________________________________________________________________
  dropout_7 (Dropout)          (None, 49, 100)           0
  _________________________________________________________________
  flatten_1 (Flatten)          (None, 4900)              0
  _________________________________________________________________
  dense_1 (Dense)              (None, 7)                 34307
  =================================================================
  Total params: 430,607
  Trainable params: 430,607
  Non-trainable params: 0
  </code></pre>
  
  <p>I'm trying to run this model for higher number of epochs. First few epochs are fine, but then it errors out:</p>
  
  <pre><code>Model] Model Compiled
  Time taken: 0:00:02.314468
  [Model] Training Started
  [Model] 100 epochs, 1000 batch size, 20.0 batches per epoch
  Epoch 1/100
  20/20 [==============================] - 5s 240ms/step - loss: 0.1631 - acc: 0.2905
  Epoch 2/100
  20/20 [==============================] - 2s 81ms/step - loss: 0.1288 - acc: 0.2455
  Epoch 3/100
  20/20 [==============================] - 1s 73ms/step - loss: 0.0952 - acc: 0.5058
  Epoch 4/100
  20/20 [==============================] - 2s 76ms/step - loss: 0.1141 - acc: 0.3288
  Epoch 5/100
  20/20 [==============================] - 2s 75ms/step - loss: 0.1064 - acc: 0.3425
  Epoch 6/100
  20/20 [==============================] - 1s 75ms/step - loss: 0.0767 - acc: 0.4213
  Epoch 7/100
  20/20 [==============================] - 1s 75ms/step - loss: 0.0635 - acc: 0.4764
  Epoch 8/100
  20/20 [==============================] - 1s 74ms/step - loss: 0.0555 - acc: 0.5274
  Epoch 9/100
  20/20 [==============================] - 1s 74ms/step - loss: 0.0544 - acc: 0.5141
  Epoch 10/100
  ...
  Epoch 61/100
  20/20 [==============================] - 1s 74ms/step - loss: 0.0506 - acc: 0.3925
  Epoch 62/100
  20/20 [==============================] - 1s 72ms/step - loss: 0.0495 - acc: 0.4323
  Epoch 63/100
  20/20 [==============================] - 1s 73ms/step - loss: 0.0495 - acc: 0.4118
  Epoch 64/100
   2/20 [==&gt;...........................] - ETA: 1s - loss: 0.0495 - acc: 0.4885Traceback (most recent call last):
    File "./run.py", line 118, in &lt;module&gt;
      main()
    File "./run.py", line 92, in main
      steps_per_epoch=steps_per_epoch)
    File "/home/sridhar/PE_CSV/alarmProj/rnn/lstm/core/model.py", line 149, in train_generator
      workers=70)
    File "/home/sridhar/PE_CSV/malenv/local/lib/python2.7/site-packages/keras/legacy/interfaces.py", line 91, in wrapper
      return func(*args, **kwargs)
    File "/home/sridhar/PE_CSV/malenv/local/lib/python2.7/site-packages/keras/engine/training.py", line 1415, in fit_generator
      initial_epoch=initial_epoch)
    File "/home/sridhar/PE_CSV/malenv/local/lib/python2.7/site-packages/keras/engine/training_generator.py", line 213, in fit_generator
      class_weight=class_weight)
    File "/home/sridhar/PE_CSV/malenv/local/lib/python2.7/site-packages/keras/engine/training.py", line 1209, in train_on_batch
      class_weight=class_weight)
    File "/home/sridhar/PE_CSV/malenv/local/lib/python2.7/site-packages/keras/engine/training.py", line 749, in _standardize_user_data
      exception_prefix='input')
    File "/home/sridhar/PE_CSV/malenv/local/lib/python2.7/site-packages/keras/engine/training_utils.py", line 127, in standardize_input_data
      'with shape ' + str(data_shape))
  ValueError: Error when checking input: expected cu_dnngru_1_input to have 3 dimensions, but got array with shape (380, 1)
  </code></pre>
  
  <p>If I reduce the number of epochs to less than that value (say epoch 64 here), I don't have any issues, but increasing the number of epochs causes the above error at some point. The exact number of epochs where it crashes seem to vary with any change to the configuration. The same issue is seen with vanilla GRU/LSTM layers.</p>
  
  <p>This is keras-2.2.2 and the model is being compiled with 70 worker threads.</p>
  
  <p>Is there something I could do to avoid this issue?</p>
  
  <p>Edit:
  Here's the relevant approximate code used:</p>
  
  <pre><code>session_conf = tf.ConfigProto(
              inter_op_parallelism_threads=multiprocessing.cpu_count(),
              intra_op_parallelism_threads=multiprocessing.cpu_count())
          sess = tf.Session(graph=tf.get_default_graph(), config=session_conf)
  K.set_session(sess)
  
  self.model.add(CuDNNGRU(
                 100,
                 input_shape=(49,7),
                 kernel_initializer='orthogonal',
                 return_sequences=true))
  self.model.add(Dropout(0.4))
  self.model.add(CuDNNGRU(
                 100,
                 input_shape=(None,None),
                 kernel_initializer='orthogonal',
                 return_sequences=true))
  self.model.add(Dropout(0.4))
  self.model.add(CuDNNGRU(
                 100,
                 input_shape=(None,None),
                 kernel_initializer='orthogonal',
                 return_sequences=true))
  self.model.add(Dropout(0.4))
  self.model.add(CuDNNGRU(
                 100,
                 input_shape=(None,None),
                 kernel_initializer='orthogonal',
                 return_sequences=true))
  self.model.add(Dropout(0.4))
  self.model.add(CuDNNGRU(
                 100,
                 input_shape=(None,None),
                 kernel_initializer='orthogonal',
                 return_sequences=true))
  self.model.add(Dropout(0.4))
  self.model.add(CuDNNGRU(
                 100,
                 input_shape=(None,None),
                 kernel_initializer='orthogonal',
                 return_sequences=true))
  self.model.add(Dropout(0.4))
  self.model.add(CuDNNGRU(
                 100,
                 input_shape=(None,None),
                 kernel_initializer='orthogonal',
                 return_sequences=true))
  self.model.add(Dropout(0.4))
  
  elf.model.add(Flatten())
  self.model.add(Dense(7, activation='relu'))
  
  sgd = SGD(lr=0.1, decay=1e-2, clipnorm=5.0)
  
  self.model.compile(
              loss='mse',
              metrics=["accuracy"],
              optimizer=sgd)
  ===================
  
   def train_generator(self, data_gen, epochs, batch_size, steps_per_epoch):
          timer = Timer()
          timer.start()
          print('[Model] Training Started')
          print('[Model] %s epochs, %s batch size, %s batches per epoch' %
                (epochs, batch_size, steps_per_epoch))
  
          save_fname = '%s/%s-e%s.h5' % (self.model_dir, dt.datetime.now()
                                         .strftime('%d%m%Y-%H%M%S'), str(epochs))
          callbacks = [
              ModelCheckpoint(
                  filepath=save_fname, monitor='loss', save_best_only=True)
          ]
          try:
              self.model.fit_generator(
                  data_gen,
                  steps_per_epoch=steps_per_epoch,
                  epochs=epochs,
                  callbacks=callbacks)
          except:
              pdb.set_trace()
  )
  
          print('[Model] Training Completed. Model saved as %s' % save_fname)
          timer.stop()
  =============
      #invoked from main function
      model.train_generator(
          data_gen=data.generate_train_batch(
              seq_len=50,
              batch_size=1000,
              normalise=false),
              epochs=100,
              batch_size=1000,
              steps_per_epoch=steps_per_epoch)
  =============
  
      def generate_train_batch(self, seq_len, batch_size, normalise):
          '''Yield a generator of training data from filename on given list of cols split for train/test'''
          i = 0
          while i &lt; (self.len_train - seq_len):
              x_batch = []
              y_batch = []
              for b in range(batch_size):
                  if i &gt;= (self.len_train - seq_len):
                      # stop-condition for a smaller final batch if data doesn't divide evenly
  
                      yield np.array(x_batch), np.array(y_batch)
                  x, y = self._next_window(i, seq_len, normalise)
                  x_batch.append(x)
                  y_batch.append(y)
                  i += 1
  
              yield np.array(x_batch), np.array(y_batch)
  =======================
  </code></pre>