title: How to check if tensorflow is using all available GPU's
body: <p>I am learning to use Tensorflow for object detection.  To speed up the training process, I have taken a AWS g3.16xlarge instance which has 4 GPUs.  I am using the following code to run training process:</p>
  
  <pre><code>export CUDA_VISIBLE_DEVICES=0,1,2,3
   python object_detection/train.py --logtostderr --pipeline_config_path=/home/ubuntu/builder/rcnn.config --train_dir=/home/ubuntu/builder/experiments/training/
  </code></pre>
  
  <p>Inside the rcnn.config - i have set the <code>batch-size = 1</code>.  During runtime I get the following output:</p>
  
  <p><strong>console output</strong></p>
  
  <pre><code>2018-11-09 07:25:50.104310: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1045] Device peer to peer matrix
  2018-11-09 07:25:50.104385: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1051] DMA: 0 1 2 3 
  2018-11-09 07:25:50.104395: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1061] 0:   Y N N N 
  2018-11-09 07:25:50.104402: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1061] 1:   N Y N N 
  2018-11-09 07:25:50.104409: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1061] 2:   N N Y N 
  2018-11-09 07:25:50.104416: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1061] 3:   N N N Y 
  2018-11-09 07:25:50.104429: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1120] Creating TensorFlow device (/device:GPU:0) -&gt; (device: 0, name: Tesla M60, pci bus id: 0000:00:1b.0, compute capability: 5.2)
  2018-11-09 07:25:50.104439: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1120] Creating TensorFlow device (/device:GPU:1) -&gt; (device: 1, name: Tesla M60, pci bus id: 0000:00:1c.0, compute capability: 5.2)
  2018-11-09 07:25:50.104446: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1120] Creating TensorFlow device (/device:GPU:2) -&gt; (device: 2, name: Tesla M60, pci bus id: 0000:00:1d.0, compute capability: 5.2)
  2018-11-09 07:25:50.104455: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1120] Creating TensorFlow device (/device:GPU:3) -&gt; (device: 3, name: Tesla M60, pci bus id: 0000:00:1e.0, compute capability: 5.2)
  </code></pre>
  
  <p>When I run <code>nvidia-smi</code>, I get the following output:
  <strong>nvidia-smi output</strong></p>
  
  <pre><code>+-----------------------------------------------------------------------------+
  | NVIDIA-SMI 375.26                 Driver Version: 375.26                    |
  |-------------------------------+----------------------+----------------------+
  | GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
  | Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
  |===============================+======================+======================|
  |   0  Tesla M60           Off  | 0000:00:1B.0     Off |                    0 |
  | N/A   52C    P0   129W / 150W |   7382MiB /  7612MiB |     92%      Default |
  +-------------------------------+----------------------+----------------------+
  |   1  Tesla M60           Off  | 0000:00:1C.0     Off |                    0 |
  | N/A   33C    P0    38W / 150W |   7237MiB /  7612MiB |      0%      Default |
  +-------------------------------+----------------------+----------------------+
  |   2  Tesla M60           Off  | 0000:00:1D.0     Off |                    0 |
  | N/A   40C    P0    38W / 150W |   7237MiB /  7612MiB |      0%      Default |
  +-------------------------------+----------------------+----------------------+
  |   3  Tesla M60           Off  | 0000:00:1E.0     Off |                    0 |
  | N/A   34C    P0    39W / 150W |   7237MiB /  7612MiB |      0%      Default |
  +-------------------------------+----------------------+----------------------+
  
  +-----------------------------------------------------------------------------+
  | Processes:                                                       GPU Memory |
  |  GPU       PID  Type  Process name                               Usage      |
  |=============================================================================|
  |    0     97860    C   python                                        7378MiB |
  |    1     97860    C   python                                        7233MiB |
  |    2     97860    C   python                                        7233MiB |
  |    3     97860    C   python                                        7233MiB |
  +-----------------------------------------------------------------------------+
  </code></pre>
  
  <p>and <code>**nvidia-smi dmon**</code> provides the following output:</p>
  
  <pre><code># gpu   pwr  temp    sm   mem   enc   dec  mclk  pclk
  # Idx     W     C     %     %     %     %   MHz   MHz
      0   158    69    90    69     0     0  2505  1177
      1    38    36     0     0     0     0  2505   556
      2    38    45     0     0     0     0  2505   556
      3    39    37     0     0     0     0  2505   556
  </code></pre>
  
  <p>I am confused with each of the output.  While I read the console output as the program is recognizing the availability of 4 different gpus, in the nvidia-smi output the volatile GPU-Util percentage is shown only for the first GPU and for the rest it is zero.  However the same table prints memory usage for all the 4 gpu's at the bottom.  And the nvidia-smi dmon prints the sm values only for first gpu and for the others it is zero.  From this <a href="https://www.microway.com/hpc-tech-tips/nvidia-smi_control-your-gpus/" rel="noreferrer">blog</a> I understand the zero in <code>dmon</code> indicates that GPU is free.  </p>
  
  <p>What I want to understand is, does the train.py utilizes all the 4 GPU's that I have in my instance. If it is not utilizing all the GPU's how do I ensure the <code>object_detection/train.py</code> of tensorflow is optimized for all the GPU's.  </p>