title: How to load only a batch of images at a time in tensorflow?
body: <p>I am reading cifar100 images from <a href="https://www.cs.toronto.edu/~kriz/cifar.html" rel="nofollow noreferrer">here</a> and I want to read the images in batches from the pickle file. With this code that I am using this line (Loaded %d examples." % num) loads all the images present in the training dataset. Then using tf.data i can read batches. But when it loads all the images my memory gets utilised and it does not even start the training. I am using something similar to <a href="https://github.com/denocris/cifar10-CNN-TensorFlow-Estimators/blob/master/cifar10_5cnn.py" rel="nofollow noreferrer">this</a>. This link uses tfrecords and I want to read cifar data using pickle. So does anyone know how I can read only batches of data from cifar100 pickle file so that my memory does not become full ?</p>
  
  <pre><code>def read_data():
      def in_data():
          all_images = []
          all_labels = []
          with open("%s%s" % ("./data/cifar-100-python/", "train"),"rb") as fo:
              dict = pickle.load(fo, encoding='latin1')
              images = np.array(dict['data'])
              labels = np.array(dict['fine_labels'])
              num = images.shape[0]
  
              # images = normalize(images)
              images = images.astype(dtype=np.float32)
              labels = labels.astype(dtype=np.int32)
              images = np.reshape(images, [num, 3, 32, 32])
              images = np.transpose(images, [0, 2, 3, 1])
  
              print("Loaded %d examples." % num)
              #print('BeforeLables: ', labels)
              labels = one_hot_encode(labels)
              #print('afterLables: ', labels)
              all_images.append(images)
              all_labels.append(labels)
              # print('SIZE:', len(all_images))
  
          all_images = np.concatenate(all_images)
          all_labels = np.concatenate(all_labels)
          self.size = len(all_images)
          img_dataset = tf.data.Dataset.from_tensor_slices(all_images).batch(2)
          label_dataset = tf.data.Dataset.from_tensor_slices(all_labels).batch(2)
  
          dataset = tf.data.Dataset.zip((img_dataset, label_dataset)).repeat(None)
  
          images, labels = dataset.make_one_shot_iterator().get_next()
          print('image Shape:',images.shape)
          return images, labels
      return in_data
  </code></pre>