title: Pytorch speed comparison - GPU slower than CPU
body: <p>I was trying to find out if GPU tensor operations are actually faster than CPU ones. So, I wrote this particular code below to implement a simple 2D addition of CPU tensors and GPU cuda tensors successively to see the speed difference:</p>
  
  <pre><code>import torch
  import time
  
  ###CPU
  start_time = time.time()
  a = torch.ones(4,4)
  for _ in range(1000000):
      a += a
  elapsed_time = time.time() - start_time
  
  print('CPU time = ',elapsed_time)
  
  ###GPU
  start_time = time.time()
  b = torch.ones(4,4).cuda()
  for _ in range(1000000):
      b += b
  elapsed_time = time.time() - start_time
  
  print('GPU time = ',elapsed_time)
  </code></pre>
  
  <p>To my surprise, the CPU time was 0.93 sec and the GPU time was as high as 63 seconds. Am I doing the cuda tensor operation properly or is the concept of cuda tensors works faster only in very highly complex operations, like in neural networks?</p>
  
  <p>Note: My GPU is NVIDIA 940MX and <code>torch.cuda.is_available()</code> call returns True.</p>