title: How to run asynchronous predictions with TensorFlow Estimator API?
body: <p>I am using the tf.estimator API to predict punctuation. I trained it with pre-processed data using TFRecords and <code>tf.train.shuffle_batch</code>. Now I want to make predictions. I can do this fine feeding static NumPy data into <code>tf.constant</code> and returning this from the <code>input_fn</code>.</p>
  
  <p>However I am working with sequence data and I need to feed one example at a time and the next input is dependent on the previous output. I also want to be able to process data input through HTTP requests.</p>
  
  <p>Every time <code>estimator.predict</code> is called it re-loads the checkpoint and recreates the entire graph. This is slow and expensive. So I need to be able to dynamically feed data to the <code>input_fn</code>.</p>
  
  <p>My current attempt is roughly this:</p>
  
  <pre><code>feature_input = tf.placeholder(tf.int32, shape=[1, MAX_SUBSEQUENCE_LEN])
  q = tf.FIFOQueue(1, tf.int32, shapes=[[1, MAX_SUBSEQUENCE_LEN]])
  enqueue_op = q.enqueue(feature_input)
  
  def input_fn():
      return q.dequeue()
  
  estimator = tf.estimator.Estimator(model_fn, model_dir=model_file)
  predictor = estimator.predict(input_fn=input_fn)
  sess = tf.Session()
  output = None
  
  while True:
      x = get_numpy_data(x, output)
      if x is None:
          break
      sess.run(enqueue_op, {feature_input: x})
      output = predictor.next()
      save_to_file(output)
  
  sess.close()
  </code></pre>
  
  <p>However I am getting the following error:
  <code>ValueError: Input graph and Layer graph are not the same: Tensor("EmbedSequence/embedding_lookup:0", shape=(1, 200, 128), dtype=float32) is not from the passed-in graph.</code></p>
  
  <p>How can I asynchronously plug data into my existing graph through an <code>input_fn</code> to get predictions one at a time?</p>