title: using keras's .flow_from_directory() on mounted s3 bucket in databricks
body: <p>I am trying to build a convolutional neural network in databricks using Spark 2.4.4 and a Scala 2.11 backend in python. I have build CNN's before but this is my first time with using Spark(databricks) and AWS s3. 
  The files in AWS are oredered like this:</p>
  
  <ul>
  <li>train_test_small/(train or test)/(0,1,2 or 3)/</li>
  </ul>
  
  <p>And then a list of images in every directory corresponding to their category(0,1,2,3)</p>
  
  <p>In order to access my files stored in the s3 bucket I mounted the bucket to databricks like this:</p>
  
  <pre><code># load in the image files
  WS_BUCKET_NAME = "sensored_bucket_name/video_topic_modelling/data/train_test_small"
  MOUNT_NAME = "train_test_small"
  dbutils.fs.mount("s3a://%s" % AWS_BUCKET_NAME, "/mnt/%s" % MOUNT_NAME)
  display(dbutils.fs.ls("/mnt/%s" % MOUNT_NAME))
  </code></pre>
  
  <p>Upon using: <code>display(dbutils.fs.mounts())</code> I can see the bucket mounted to:</p>
  
  <p><code>MountInfo(mountPoint='/mnt/train_test_small', source='sensored_bucket_name/video_topic_modelling/data/train_test_small', encryptionType='')</code></p>
  
  <p>I then try to access this mounted directory through keras's flow_from_directory() module using the following piece of code:</p>
  
  <pre><code># create extra partition of the training data as a validation set
  train_datagen=ImageDataGenerator(preprocessing_function=preprocess_input, validation_split=0) #included in our dependencies
  
  # set scaling to most common shapes
  train_generator=train_datagen.flow_from_directory('/mnt/train_test_small',
                                                   target_size=(320, 240),
                                                   color_mode='rgb',
                                                   batch_size=96,
                                                   class_mode='categorical',
                                                   subset='training')
                                                   #shuffle=True)
  validation_generator=train_datagen.flow_from_directory('/mnt/train_test_small',
                                                   target_size=(320, 240),
                                                   color_mode='rgb',
                                                   batch_size=96,
                                                   class_mode='categorical',
                                                   subset='validation')
  </code></pre>
  
  <p>However this gives me the following error:</p>
  
  <p><code>FileNotFoundError: [Errno 2] No such file or directory: '/mnt/train_test_small/train/'</code></p>
  
  <p>I tried to figure this out using keras and databricks documentation but got no further. My best guess at the moment right now is that the keras flow_from_directory() is unable to detect mounted directories but I am not sure. </p>
  
  <p>Anyone out there who does know how to apply the .flow_from_directory() module on a s3 mounted directory in databricks or who knows a good alternative? Help would be much appreciated!</p>