title: Multi GPU Training in Tensorflow (Data Parallelism) when Using feed_dict
body: <p>I would like to use multiple GPUs to train my Tensorflow model taking advantage of data parallelism.</p>
  
  <p>I am currently training a Tensorflow model using the following approach:</p>
  
  <pre><code>x_ = tf.placeholder(...)
  y_ = tf.placeholder(...)
  y = model(x_)
  loss = tf.losses.sparse_softmax_cross_entropy(labels=y_, logits=y)
  optimizer = tf.train.AdamOptimizer()
  train_op = tf.contrib.training.create_train_op(loss, optimizer)
  for i in epochs:
     for b in data:
        _ = sess.run(train_op, feed_dict={x_: b.x, y_: b.y})
  </code></pre>
  
  <p>I would like to take advantage of multiple GPUs to train this model in a data parallelize manner. i.e. I would like to split my batches in half and run each half batch on one of my two GPUs.</p>
  
  <p><a href="https://github.com/tensorflow/models/blob/master/tutorials/image/cifar10/cifar10_multi_gpu_train.py" rel="noreferrer">cifar10_multi_gpu_train</a> seems to provide a good example of creating a loss that draws from graphs running on multiple GPUs, but I haven't found a good examples of doing this style of training when using <code>feed_dict</code> and <code>placeholder</code> as opposed to a data loader queue.</p>
  
  <p><em>UPDATE</em></p>
  
  <p>Seems like: <a href="https://timsainb.github.io/multi-gpu-vae-gan-in-tensorflow.html" rel="noreferrer">https://timsainb.github.io/multi-gpu-vae-gan-in-tensorflow.html</a> might provide a good example. They seem to pull in <code>average_gradients</code> from <code>cifar10_multi_gpu_train.py</code> and create one placeholder which they then slice into for each of the GPUs. 
  I think you also need to split <code>create_train_op</code> into three stages: <code>compute_gradients</code>, <code>average_gradients</code> and then <code>apply_gradients</code>.</p>