title: Keras & Tensorflow GPU Out of Memory on Large Image Data
body: <p>I'm building an image classification system with Keras, Tensorflow GPU backend and CUDA 9.1, running on Ubuntu  18.04.</p>
  
  <p>I'm using a very large image data set with 1.2 million images, 15k classes, and is 335 GB in size.</p>
  
  <p>I can train my network on 90,000 images with no problems.  However, when I scale up and use the entire data set of 1.2 million images I get the error shown below, which I believe have to do with running out of memory.</p>
  
  <p>I'm using GeForce GTX 1080 with 11GB memory, and I have 128GB of RAM, 300GB of swap file and AMD Threadripper 1950X with 16 cores.</p>
  
  <p>I followed the advice given to solve similar problems.  I'm now using <a href="https://stackoverflow.com/questions/37313818/tensorflow-dst-tensor-is-not-initialized">smaller batch size of 10</a> or <a href="https://github.com/aymericdamien/TensorFlow-Examples/issues/38" rel="nofollow noreferrer">even smaller</a>, and a <a href="https://github.com/keras-team/keras/issues/5199" rel="nofollow noreferrer">smaller dense inner layer</a> of 256, and I'm still getting the same error shown below before the first training epoch begins.</p>
  
  <p><strong>[Update]:</strong> I found out that The memory error happens during the VGG16 <code>predict_generator</code> call, even before my network is built or trained.  See code below.</p>
  
  <p>First, warnings and Errors:</p>
  
  <pre><code>2018-05-19 20:24:01.255788: E tensorflow/stream_executor/cuda/cuda_driver.cc:967] failed to alloc 5635855360 bytes on host: CUresult(304)
  2018-05-19 20:24:01.255850: W ./tensorflow/core/common_runtime/gpu/pool_allocator.h:195] could not allocate pinned host memory of size: 5635855360
  </code></pre>
  
  <p>Then exceptions:</p>
  
  <pre><code>2018-05-19 13:56:40.472404: I tensorflow/core/common_runtime/bfc_allocator.cc:680] Stats: 
  Limit:                 68719476736
  InUse:                 15548829696
  MaxInUse:              15548829696
  NumAllocs:                   15542
  MaxAllocSize:             16777216
  
  2018-05-19 13:56:40.472563: W tensorflow/core/common_runtime/bfc_allocator.cc:279] ****************************************************************************************************
  Traceback (most recent call last):
    File "/home/welshamy/tools/anaconda/3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1322, in _do_call
      return fn(*args)
    File "/home/welshamy/tools/anaconda/3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1307, in _run_fn
      options, feed_dict, fetch_list, target_list, run_metadata)
    File "/home/welshamy/tools/anaconda/3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1409, in _call_tf_sessionrun
      run_metadata)
  tensorflow.python.framework.errors_impl.InternalError: Dst tensor is not initialized.
       [[Node: block5_pool/MaxPool/_159 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_133_block5_pool/MaxPool", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]
  
  During handling of the above exception, another exception occurred:
  
  Traceback (most recent call last):
    File "bottleneck.py", line 37, in &lt;module&gt;
      bottleneck_features_train = model_vgg.predict_generator(train_generator_bottleneck)
    File "/home/welshamy/tools/anaconda/3/lib/python3.6/site-packages/keras/legacy/interfaces.py", line 91, in wrapper
      return func(*args, **kwargs)
    File "/home/welshamy/tools/anaconda/3/lib/python3.6/site-packages/keras/engine/training.py", line 2510, in predict_generator
      outs = self.predict_on_batch(x)
    File "/home/welshamy/tools/anaconda/3/lib/python3.6/site-packages/keras/engine/training.py", line 1945, in predict_on_batch
      outputs = self.predict_function(ins)
    File "/home/welshamy/tools/anaconda/3/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py", line 2478, in __call__
      **self.session_kwargs)
    File "/home/welshamy/tools/anaconda/3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 900, in run
      run_metadata_ptr)
    File "/home/welshamy/tools/anaconda/3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1135, in _run
      feed_dict_tensor, options, run_metadata)
    File "/home/welshamy/tools/anaconda/3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1316, in _do_run
      run_metadata)
    File "/home/welshamy/tools/anaconda/3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1335, in _do_call
      raise type(e)(node_def, op, message)
  tensorflow.python.framework.errors_impl.InternalError: Dst tensor is not initialized.
       [[Node: block5_pool/MaxPool/_159 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_133_block5_pool/MaxPool", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]
  </code></pre>
  
  <p>Here is my code:</p>
  
  <pre class="lang-python prettyprint-override"><code>import numpy as np
  from keras.callbacks import EarlyStopping, ModelCheckpoint
  from keras.layers import Dropout, Flatten, Dense
  from keras.models import Sequential
  from keras.preprocessing.image import ImageDataGenerator
  from keras import applications
  from keras.utils.np_utils import to_categorical
  import matplotlib.pyplot as plt
  
  # Dimensions of our images.
  img_width, img_height = 224, 224
  
  train_data_dir = './train_sample'
  
  epochs = 100
  batch_size = 10
  
  # Data preprocessing
  # Pixel values rescaling from [0, 255] to [0, 1] interval
  datagen = ImageDataGenerator(rescale=1. / 255)
  
  # Retrieve images and their classes for training set.
  train_generator_bottleneck = datagen.flow_from_directory(
      train_data_dir,
      target_size=(img_width, img_height),
      batch_size=batch_size,
      class_mode=None,
      shuffle=False)
  
  
  num_classes = len(train_generator_bottleneck.class_indices)
  
  model_vgg = applications.VGG16(include_top=False, weights='imagenet')
  
  bottleneck_features_train = model_vgg.predict_generator(train_generator_bottleneck)
  np.save('../models/bottleneck_features_train.npy', bottleneck_features_train)
  
  train_data = np.load('../models/bottleneck_features_train.npy')
  train_labels = to_categorical(train_generator_bottleneck.classes, num_classes=num_classes)
  
  
  model_top = Sequential()
  model_top.add(Flatten(input_shape=train_data.shape[1:]))
  model_top.add(Dense(256, activation='relu'))
  model_top.add(Dropout(0.5))
  model_top.add(Dense(num_classes, activation='softmax'))
  
  model_top.compile(loss='categorical_crossentropy',
                    optimizer='rmsprop',
                    metrics=['accuracy'])
  
  # Model saving callback
  checkpointer = ModelCheckpoint(filepath='../models/bottleneck_features.h5', monitor='val_acc', verbose=1,
                                 save_best_only=True)
  
  # Early stopping
  early_stopping = EarlyStopping(monitor='val_acc', verbose=1, patience=5)
  
  history = model_top.fit(
      train_data,
      train_labels,
      verbose=2,
      epochs=epochs,
      batch_size=batch_size,
      callbacks=[checkpointer, early_stopping],
      validation_split=0.3)
  </code></pre>