title: Dealing with big training data
body: <p>I am writing to ask the principle of how to feed a big training data to a tensor flow model. My training data is hosted in csv file(s), and basically I am using the below code to load data in queue. </p>
  
  <pre><code>filename_queue = tf.train.string_input_producer([...])
  reader = tf.TextLineReader()
  _, line = reader.read(filename_queue)
  
  line = tf.decode_csv(line, record_defaults=default)
  label_batch, feature_batch = tf.train.shuffle_batch([label, feature], batch_size=batch_size, capacity=512, min_after_dequeue=256,  num_threads=8)
  </code></pre>
  
  <p>My experiment as below:</p>
  
  <ul>
  <li>Exp#1: a single file which 140G of size. my program got "killed", guessing OOM.</li>
  <li>Exp#2: split the big file into 340 small ones. program worked, but GPU utilized almost => 0%, from tensor board items in batch queue getting smaller. </li>
  </ul>
  
  <p>I think in #2 loading data into queue is bottleneck, but I have already using 8 threads to load data. Although a sample in training data is as big as 224*224*3 features, I don't think it should be so slow just loading 512 of them into 16GB memory.</p>
  
  <p>Any one can shed some lights on it? What should I do next? convert csv to binary?    </p>