title: Learning rate doesn't change for AdamOptimizer in TensorFlow
body: <p>I would like to see how the learning rate changes during training (print it out or create a summary and visualize it in tensorboard).</p>
  
  <p>Here is a code snippet from what I have so far:</p>
  
  <pre class="lang-py prettyprint-override"><code>optimizer = tf.train.AdamOptimizer(1e-3)
  grads_and_vars = optimizer.compute_gradients(loss)
  train_op = optimizer.apply_gradients(grads_and_vars, global_step=global_step)
  
  sess.run(tf.initialize_all_variables())
  
  for i in range(0, 10000):
     sess.run(train_op)
     print sess.run(optimizer._lr_t)
  </code></pre>
  
  <p>If I run the code I constantly get the initial learning rate (1e-3) i.e. I see no change.</p>
  
  <p>What is a correct way for getting the learning rate at every step?</p>
  
  <p>I would like to add that <a href="https://stackoverflow.com/questions/36990476/getting-the-current-learning-rate-from-a-tf-train-adamoptimizer">this question</a> is really similar to mine. However, I cannot post my findings in the comment section there since I do not have enough rep.</p>