title: TPU slower than GPU?
body: <p>I just tried using TPU in Google Colab and I want to see how much TPU is faster than GPU.  I got surprisingly the opposite result.</p>
  
  <p>The following is the NN.  </p>
  
  <pre><code>  random_image = tf.random_normal((100, 100, 100, 3))
    result = tf.layers.conv2d(random_image, 32, 7)
    result = tf.reduce_sum(result)
  </code></pre>
  
  <p>Performance results:</p>
  
  <pre><code>CPU: 8s
  GPU: 0.18s
  TPU: 0.50s
  </code></pre>
  
  <p>I wonder why.... The complete code for TPU is as follows:</p>
  
  <pre><code>def calc():
    random_image = tf.random_normal((100, 100, 100, 3))
    result = tf.layers.conv2d(random_image, 32, 7)
    result = tf.reduce_sum(result)
    return result
  
  tpu_ops = tf.contrib.tpu.batch_parallel(calc, [], num_shards=8)
  
  session = tf.Session(tpu_address)
  try:
    print('Initializing global variables...')
    session.run(tf.global_variables_initializer())
    print('Warming up...')
    session.run(tf.contrib.tpu.initialize_system())
    print('Profiling')
    start = time.time()
    session.run(tpu_ops)
    end = time.time()
    elapsed = end - start
    print(elapsed)
  finally:
    session.run(tf.contrib.tpu.shutdown_system())
    session.close()
  </code></pre>