title: Tensorflow image input pipeline for large datasets: Loading multiple consecutive images
body: <p>I am trying to build a somewhat complicated input pipeline for Tensorflow.
  The dataset consists of 257 days of images with around 5000 images each day (the images are taken roughly every 8 seconds and ordered temporally).
  They are encoded in jpeg. I need to be able to access them in order (temporally) within each day. However, I also need to be able to create shuffled batches (randomized both across days and within a day). Furthermore, one sample might consist of 2 or 3 consecutive images with the labels of the last image respectively. Additionally I should be able to add a stride or a padding between the images in one sample (for example, I have a sample consisting of 2 images that are 16 seconds apart instead of only 8)</p>
  
  <p>My first approach was to create TFRecord files with the decoded images and labels (consisting of 23 float32). Of course this would have blown up the memory requirements by around 600GB.</p>
  
  <p>Now I created TFRecord files for each day (257). They encode the path to an image along with its labels (path,label1 .. label23). Within one TFRecord files, the days are encoded in order, starting from 6:30:00 am up to around 18:00:00 pm.</p>
  
  <p>Below you see my current (simplified) solution. I'm using Data Readers (TFRecordReader) and queue runners. I manage to load batches of images and their respective labels in order (when not using batch_shuffle) and randomly (when using batch_shuffle).</p>
  
  <p><strong>However I encounter the following problems:</strong></p>
  
  <ul>
  <li>Since TFRecord files are read sequentially, I need to have a large amount of samples in the queue to allow for enough randomization (batch shuffle) within each day. However, more then 1500 samples in the queue exceed my memory.</li>
  <li>When I input several TFRecord files (one for each day) into the tf.train.string_input_producer, there does not seem to be a randomization across days. Basically the samples are always from the same day. </li>
  </ul>
  
  <p><strong>Questions:</strong></p>
  
  <ul>
  <li><p>How can I read for example two consecutive images per sample? Should I just call the tfreader.read function twice? Is it guaranteed to read the next image (chronologically saved in tfrecords file, as mentioned above) or will there be issues due to multiple queue runners? In the end i want to have batches of (Tensor(image_06:30:00, image_06:30:08),Tensor(labels of image_06:30:08))</p></li>
  <li><p>How can I ensure that the tf.train.string_input_producer randomly switches across the different TFRecord files in the input list? While preserving the order of the image when creating the samples. For example when I want to read 2 images for one sample they should be read from the same day (the same tfrecords file)</p></li>
  <li><p>How can I add strides larger than 1? Essentially I want to have every 2nd or third image in one sample. for example: (Tensor(image_06:30:00, image_06:30:16),Tensor(labels of image_06:30:16)) instead of image_06:30:08</p></li>
  <li><p>How should I handle the problem with exceeding memory usage? Should I decode the image when I use it, instead of creating large samples with the decoded image tensors and then put them into the Queue? So basically I create batches of samples that only contain the path to an image and the labels. then I decode the image while training/evaluating</p></li>
  </ul>
  
  <p><strong>Current solution (simplified, does not compile!):</strong></p>
  
  <pre><code>IMAGE_HEIGHT=1566
  IMAGE_WIDTH=1566
  IMAGE_CHANNELS=3
  
  def decode_single_tfrecord(filename_queue,feature_dict,name):
    tfreader = tf.TFRecordReader()
    key,serialized_value = tfreader.read(filename_queue,name=name)
    features = tf.parse_single_example(serialized_value,features=feature_dict)
    return key,features
  
  
  def get_example(decoded_features,feature_list,img_path_key='image_path'):
    labels = list()
    for feature in feature_list:
      labels.append(decoded_features[feature])
  
    label_tensor = tf.stack(labels)
  
    image_path = decoded_features[img_path_key]
    with tf.name_scope('decode_image',[image_path],None):
      image_file = tf.read_file(image_path)
      image_file = tf.image.decode_jpeg(image_file,channels=IMAGE_CHANNELS)
  
    image_tensor = tf.reshape(image_file,(IMAGE_HEIGHT,IMAGE_WIDTH,IMAGE_CHANNELS))
    return image_tensor,label_tensor
  
  
  #############################
  #Encodes the label key in the tfrecord file
  feature_list=['VF','IRR0','IRR1','IRR2','IRR3','IRR4','IRR5','IRR6','IRR7','IRR8','IRR9',/
  'IRR10','MPC0','MPC1','MPC2','MPC3','MPC4','MPC5','MPC6','MPC7','MPC8','MPC9','MPC10']
  features={}
  for f in feature_list:
    features[f] = tf.FixedLenFeature([], tf.float32)
  
  #Path to image in tfrecord file
  features['image_path'] = tf.FixedLenFeature([], tf.string)
  
  
  #GRAPH#######################
  filename_queue = tf.train.string_input_producer(["paths to tfrecords"])
  tf_key,decoded_f = decode_single_tfrecord(filename_queue,features, name="tf_records_reader")
  
  image,label = get_example(decoded_f,feature_list)
  
  #simply calls tf.train.shuffle_batch([image,label],..)
  imageBatch,labelBatch =   create_shuffled_data_batch(image,label,) 
  
  
  init_op = tf.group(tf.global_variables_initializer(),tf.local_variables_initializer())
  with tf.Session() as sess:
     sess.run(init_op)
     coord = tf.train.Coordinator()
     threads = tf.train.start_queue_runners(coord=coord)
  
     for i in range(100):
         f,l = sess.run([imageBatch,labelBatch])
  
     coord.request_stop()
     coord.join(threads)
  </code></pre>