title: Tensorflow Timeline Metadata coming back NaN
body: <p>I have a TensorFlow program designed to help me informally benchmark a GPU I am using and, more importantly, understand the profiling tools that come with TensorFlow.  The code does nothing but set up two placeholders for a matrix, and a matmul op, then invokes a session to populate the placeholders and execute the graph ten times. </p>
  
  <p>Here is the code:</p>
  
  <pre><code>n = 10240
  iter = 10
  
  tf.reset_default_graph()
  graph = tf.Graph()
  with graph.as_default():
      with tf.device("/gpu:0"):
          matrix1 = tf.placeholder(tf.float32, [n, n], name="Matrix_One")
          matrix2 = tf.placeholder(tf.float32, [n, n], name="Matrix_Two")
          product = tf.matmul(matrix1, matrix2, name = "Matrix_Multiply")
  
  date = datetime.now()
  cwd = os.getcwd()
  LogBase = cwd + "/benchmarks2/"
  LogPath = LogBase + date.strftime("%Y%m%d-%H%M%S") + "/"
  print(LogPath)
  
  with tf.Session(graph=graph) as session:
      tf.global_variables_initializer().run()
  
      run_options = tf.RunOptions(trace_level=tf.RunOptions.FULL_TRACE)
      run_metadata = tf.RunMetadata()
  
      writer = tf.summary.FileWriter(LogPath, session.graph)
      for i in range(iter):
          m1 = np.random.rand(n, n)
          m2 = np.random.rand(n, n)
          feed_dict = { matrix1 : m1,matrix2 : m2}
          p = session.run( [product], feed_dict=feed_dict, options=run_options, run_metadata=run_metadata)
          writer.add_run_metadata(run_metadata, 'step%2d' % i)
  
      tf.summary.FileWriter(LogPath, graph).close()
  </code></pre>
  
  <p>Two peculiarities emerge when I look at the tensorboard graph:</p>
  
  <ul>
  <li>Note that I run ten iterations, but the tensorboard only shows eight steps, which seems odd</li>
  <li>Most importantly, I never see anything but NaN for the compute time or the memory, as shown below.  Note the Matrix Multiple operation is selected. </li>
  </ul>
  
  <p>Why is this happening, and how can I fix it?
  <a href="https://i.stack.imgur.com/q3f19.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/q3f19.png" alt="enter image description here"></a></p>