title: Resource exhausted Error: Keras using Tensorflow backend
body: <p>I created a docker image with Tensorflow and Keras and when I create a container it works fine but only for the first time. The second time when I try to run the same code it gives me an error:</p>
  
  <pre><code>2017-10-04 13:57:25.359384: I tensorflow/core/common_runtime/bfc_allocator.cc:660] Bin for 1.12MiB was 1.00MiB, Chunk State:
  2017-10-04 13:57:25.359402: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1041ac00000 of size 1280
  2017-10-04 13:57:25.359420: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1041ac00500 of size 256
  2017-10-04 13:57:25.359435: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1041ac00600 of size 256
  2017-10-04 13:57:25.359463: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1041ac00700 of size 256
  2017-10-04 13:57:25.359478: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1041ac00800 of size 256
  2017-10-04 13:57:25.359492: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1041ac00900 of size 512
  2017-10-04 13:57:25.359507: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1041ac00b00 of size 256
  2017-10-04 13:57:25.359521: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1041ac00c00 of size 256
  2017-10-04 13:57:25.359538: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1041ac00d00 of size 1024
  2017-10-04 13:57:25.359551: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1041ac01100 of size 256
  2017-10-04 13:57:25.359565: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1041ac01200 of size 256
  2017-10-04 13:57:25.359581: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1041ac01300 of size 2048
  2017-10-04 13:57:25.359595: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1041ac01b00 of size 256
  2017-10-04 13:57:25.359608: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1041ac01c00 of size 256
  2017-10-04 13:57:25.359624: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1041ac01d00 of size 256
  2017-10-04 13:57:25.359638: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1041ac01e00 of size 256
  2017-10-04 13:57:25.359653: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1041ac01f00 of size 256
  2017-10-04 13:57:25.359666: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1041ac02000 of size 256
  2017-10-04 13:57:25.359679: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1041ac02100 of size 256
  2017-10-04 13:57:25.359695: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1041ac02200 of size 256
  2017-10-04 13:57:25.359708: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1041ac02300 of size 256
  2017-10-04 13:57:25.359722: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1041ac03f00 of size 256
  2017-10-04 13:57:25.359736: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1041ac28000 of size 256
  2017-10-04 13:57:25.359749: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1041ac70100 of size 512
  2017-10-04 13:57:25.359765: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1041ad00300 of size 512
  2017-10-04 13:57:25.359780: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1041ad00500 of size 1179648
  2017-10-04 13:57:25.359793: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1041ae20500 of size 1024
  2017-10-04 13:57:25.359807: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Chunk at 0x1041ae20900 of size 3405568
  2017-10-04 13:57:25.359821: I tensorflow/core/common_runtime/bfc_allocator.cc:687] Free at 0x1041ac02400 of size 6912
  2017-10-04 13:57:25.359835: I tensorflow/core/common_runtime/bfc_allocator.cc:687] Free at 0x1041ac04000 of size 147456
  2017-10-04 13:57:25.359849: I tensorflow/core/common_runtime/bfc_allocator.cc:687] Free at 0x1041ac28100 of size 294912
  2017-10-04 13:57:25.359863: I tensorflow/core/common_runtime/bfc_allocator.cc:687] Free at 0x1041ac70300 of size 589824
  2017-10-04 13:57:25.359878: I tensorflow/core/common_runtime/bfc_allocator.cc:693]      Summary of in-use Chunks by size:
  2017-10-04 13:57:25.359896: I tensorflow/core/common_runtime/bfc_allocator.cc:696] 19 Chunks of size 256 totalling 4.8KiB
  2017-10-04 13:57:25.359913: I tensorflow/core/common_runtime/bfc_allocator.cc:696] 3 Chunks of size 512 totalling 1.5KiB
  2017-10-04 13:57:25.359929: I tensorflow/core/common_runtime/bfc_allocator.cc:696] 2 Chunks of size 1024 totalling 2.0KiB
  2017-10-04 13:57:25.359944: I tensorflow/core/common_runtime/bfc_allocator.cc:696] 1 Chunks of size 1280 totalling 1.2KiB
  2017-10-04 13:57:25.359960: I tensorflow/core/common_runtime/bfc_allocator.cc:696] 1 Chunks of size 2048 totalling 2.0KiB
  2017-10-04 13:57:25.359976: I tensorflow/core/common_runtime/bfc_allocator.cc:696] 1 Chunks of size 1179648 totalling 1.12MiB
  2017-10-04 13:57:25.359992: I tensorflow/core/common_runtime/bfc_allocator.cc:696] 1 Chunks of size 3405568 totalling 3.25MiB
  2017-10-04 13:57:25.360007: I tensorflow/core/common_runtime/bfc_allocator.cc:700] Sum Total of in-use chunks: 4.38MiB
  2017-10-04 13:57:25.360028: I tensorflow/core/common_runtime/bfc_allocator.cc:702] Stats:
  Limit:                     5636096
  InUse:                     4596992
  MaxInUse:                  5636096
  NumAllocs:                      32
  MaxAllocSize:              3405568
  
  2017-10-04 13:57:25.360052: W tensorflow/core/common_runtime/bfc_allocator.cc:277] *_*_____*_________****************************************************************xxxxxxxxxxxxxxxxxx
  2017-10-04 13:57:25.360084: W tensorflow/core/framework/op_kernel.cc:1192] Resource exhausted: OOM when allocating tensor with shape[3,3,128,256]
  </code></pre>
  
  <p>I think it has to do with the session and memory leaked but I have no idea how to solve it. Every time I have to delete the container and create a new one just to run one file.</p>
  
  <p>Code:</p>
  
  <pre><code>import numpy
  from keras.models import Model, load_model
  from keras.preprocessing.image import ImageDataGenerator
  from keras.layers import Flatten, Dense, Input, concatenate
  from keras.callbacks import ModelCheckpoint, CSVLogger
  from sklearn.metrics import confusion_matrix
  
  '''
  Model Parameters
  '''
  # dimensions of our images.
  model_name = 'Fuse_Net'
  img_width, img_height = 224, 224 # Resolution of inputs
  channel = 3
  nb_train_samples = 29799
  nb_validation_samples = 4428
  epochs = 20
  batch_size = 40
  
  # Dataset for Stream 1
  train_data_dir_1 = '2_Dataset/frontalization/training/'
  validation_data_dir_1 = '2_Dataset/frontalization/validation/'
  # Dataset for Stream 2
  train_data_dir_2 = '2_Dataset/patches/training/'
  validation_data_dir_2 = '2_Dataset/patches/validation/'
  
  # Custom parameters
  nb_class = 12
  hidden_dim = 512
  
  '''
  Load the models
  '''
  vgg_model_frontalization = load_model('weights/Stream_1_Frontalization.hdf5')
  vgg_model_patch = load_model('weights/Stream_2_Patches.hdf5')
  
  '''
  Customize the model
  '''
  stream_1 = vgg_model_frontalization.get_layer('pool5').output
  stream_1 = Flatten(name='flatten-1')(stream_1)
  
  for layer in vgg_model_frontalization.layers:
     layer.name = layer.name + "-model_frontalization"
  
  stream_2 = vgg_model_patch.get_layer('pool5').output
  stream_2 = Flatten(name='flatten-2')(stream_2)
  
  for layer in vgg_model_frontalization.layers:
     layer.name = layer.name + "-model_patches"
  
  # fuse_layer = stream_1 + stream_2
  fuse_layer = concatenate([stream_1, stream_2])
  
  fuse_layer = Dense(hidden_dim, activation='relu', name='fc6')(fuse_layer)
  fuse_layer = Dense(hidden_dim, activation='relu', name='fc7')(fuse_layer)
  fuse_layer = Dense(hidden_dim, activation='relu', name='fc8')(fuse_layer)
  out = Dense(nb_class, activation='softmax', name='fc9')(fuse_layer)
  
  custom_vgg_model = Model(
      [vgg_model_frontalization.input, vgg_model_patch.input],
      out)
  
  
  '''
  Dataset Generators
  '''
  train_datagen = ImageDataGenerator(
      rescale=1. / 224,
      shear_range=0.2,
      zoom_range=0.2,
      horizontal_flip=True)
  
  test_datagen = ImageDataGenerator(
      rescale=1. / 224)
  
  # Datagen
  def obtain_datagen(datagen, train_path, seed_number=7):
      return datagen.flow_from_directory(
                  train_path,
                  target_size=(img_height, img_width),
                  batch_size=batch_size,
                  seed=seed_number,
                  class_mode='binary')
  
  # Training data generators
  train_generator_1 = obtain_datagen(train_datagen, train_data_dir_1)
  train_generator_2 = obtain_datagen(train_datagen, train_data_dir_2)
  
  # Validation data generators
  validation_generator_1 = obtain_datagen(test_datagen, validation_data_dir_1)
  validation_generator_2 = obtain_datagen(test_datagen, validation_data_dir_2)
  
  # Yield for data generators
  def generate_data_generator_for_two_images(genX1, genX2):
      while True:
              X1i = genX1.next()
              X2i = genX2 .next()
              yield [X1i[0], X2i[0] ], X1i[1]
  
  # Yield for data generators
  dataset_train_gen = generate_data_generator_for_two_images(train_generator_1, train_generator_2)
  dataset_val_gen = generate_data_generator_for_two_images(validation_generator_1, validation_generator_2)
  
  
  '''
  Fine-tune the model
  '''
  # Freeze previous layers
  for i, layer in enumerate(vgg_model_frontalization.layers):
      layer.trainable = False
  for i, layer in enumerate(vgg_model_patch.layers):
      layer.trainable = False 
  
  # Compile the model
  custom_vgg_model.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])
  
  
  '''
  Callbacks
  '''
  checkpointer = ModelCheckpoint(filepath='weights/'+model_name+'.epoch-{epoch:02d}_val-accu-{val_acc:.2f}.hdf5', verbose=1, save_best_only=True)
  csv_logger = CSVLogger('logs/'+model_name+'.log', separator=',', append=False)
  
  
  '''
  Train the model
  '''
  print('*************************************\nFine-tuning the model \n*************************************')
  custom_vgg_model.fit_generator(
      dataset_train_gen,
      steps_per_epoch=nb_train_samples,
      epochs=epochs,
      validation_data=dataset_val_gen,
      validation_steps=nb_validation_samples,
      callbacks=[checkpointer, csv_logger])
  </code></pre>