title: batch normalization parameters update in multi gpu tensorflow code?
body: <p>I have written a multi gpu Cnn code</p>
  
  <p>In <a href="https://github.com/tensorflow/models/blob/master/research/inception/inception/inception_train.py" rel="nofollow noreferrer">this link</a> 
  they commented in line 249</p>
  
  <blockquote>
    <p># Retain the Batch Normalization updates operations only from the
              # final tower. Ideally, we should grab the updates from all towers
              # but these stats accumulate extremely fast so we can ignore the
              # other stats from the other towers without significant detriment.</p>
  </blockquote>
  
  <p>but this code  of updating batch norm(in line 253) :</p>
  
  <pre><code>with tf.device('/gpu:%d' % i):
  .
  .
  .
    batchnorm_updates = 
    tf.get_collection(slim.ops.UPDATE_OPS_COLLECTION,scope)
  </code></pre>
  
  <p>is done for all towers(GPUs)
   so what is the right place of this update? </p>