title: How to keep tensorflow session open between predictions? Loading from SavedModel
body: <p>I trained a tensorflow model that i'd like to run predictions on from numpy arrays. This is for image processing within videos. I will pass the images to the model as they happen. Not every frame is passed.</p>
  
  <p>I <a href="https://stackoverflow.com/questions/43667018/locally-load-saved-tensorflow-model-pb-from-google-cloud-machine-learning-engin">reload</a> my SavedModel within a session like so</p>
  
  <pre><code>def run(self):                
      with tf.Session(graph=tf.Graph()) as sess:
          tf.saved_model.loader.load(sess,
                      [tf.saved_model.tag_constants.SERVING], "model")
  </code></pre>
  
  <p>My code works perfectly if I pass a list of images (self.tfimages) to the prediction. Condensed to:</p>
  
  <pre><code>    softmax_tensor = sess.graph.get_tensor_by_name('final_ops/softmax:0')
      predictions = sess.run(softmax_tensor, {'Placeholder:0': self.tfimages})
  </code></pre>
  
  <p>But i won't have all the images at once. Do I really have to reload the model from file each time (takes 2+ minutes).</p>
  
  <p>I thought to do something like this</p>
  
  <pre><code>class tensorflow_model:
  def __init__(self):                
      with tf.Session(graph=tf.Graph()) as self.sess:
          tf.saved_model.loader.load(self.sess,
                      [tf.saved_model.tag_constants.SERVING], "model")
  def predict(self):
  
          # Feed the image_data as input to the graph and get first prediction
          softmax_tensor = self.sess.graph.get_tensor_by_name('final_ops/softmax:0')
  
          predictions = self.sess.run(softmax_tensor, {'Placeholder:0': self.tfimages})
  </code></pre>
  
  <p>but that yields</p>
  
  <p>builtins.RuntimeError: Attempted to use a closed Session</p>
  
  <p>Is there a way to keep a session open, or perhaps load SavedModel independent of a session?</p>
  
  <p>EDIT
  I tried the first answer to create a session in two steps:</p>
  
  <pre><code>sess=tf.Session(graph=tf.Graph())
  sess
  &lt;tensorflow.python.client.session.Session object at 0x0000021ACBB62EF0&gt;
  tf.saved_model.loader.load(sess,[tf.saved_model.tag_constants.SERVING], "model")
  Traceback (most recent call last):
    Debug Probe, prompt 138, line 1
    File "C:\Program Files\Python35\Lib\site-packages\tensorflow\python\saved_model\loader_impl.py", line 222, in load
      saver.restore(sess, variables_path)
    File "C:\Program Files\Python35\Lib\site-packages\tensorflow\python\training\saver.py", line 1428, in restore
      {self.saver_def.filename_tensor_name: save_path})
    File "C:\Program Files\Python35\Lib\site-packages\tensorflow\python\client\session.py", line 774, in run
      run_metadata_ptr)
    File "C:\Program Files\Python35\Lib\site-packages\tensorflow\python\client\session.py", line 905, in _run
      raise RuntimeError('The Session graph is empty.  Add operations to the '
  builtins.RuntimeError: The Session graph is empty.  Add operations to the graph before calling run().
  </code></pre>
  
  <p>Whereas </p>
  
  <pre><code>with tf.Session(graph=tf.Graph()) as sess:
      tf.saved_model.loader.load(sess,[tf.saved_model.tag_constants.SERVING], "model")
  </code></pre>
  
  <p>executes without error.</p>
  
  <p>As for the second idea of passing sess as a variable to class, which is a good one. This works:</p>
  
  <pre><code>with tf.Session(graph=tf.Graph()) as sess:
      tf.saved_model.loader.load(sess,[tf.saved_model.tag_constants.SERVING], "model")
      tensorflow_instance=tensorflow(read_from="file")
      tensorflow_instance.predict(sess)
  </code></pre>
  
  <p>But this doesn't</p>
  
  <pre><code>sess=tf.Session(graph=tf.Graph())
  tf.saved_model.loader.load(sess,[tf.saved_model.tag_constants.SERVING], "model")
  tensorflow_instance=tensorflow(read_from="file")
  tensorflow_instance.predict(sess)
  </code></pre>
  
  <p>It would be pretty awkward to wrap my program into the with as sess statement.</p>
  
  <p>Full code:</p>
  
  <pre><code>import tensorflow as tf
  import sys
  from google.protobuf import text_format
  from tensorflow.core.framework import graph_pb2
  import os
  import glob
  
  class tensorflow:    
  
  def __init__(self,read_from):
  
      #frames to be analyzed
      self.tfimages=[]    
  
      find_photos=glob.glob("*.jpg")
  
      # Read in the image_data
      if read_from=="file":
          for x in find_photos:
              image_data = tf.gfile.FastGFile(x, 'rb').read()    
              self.tfimages.append(image_data)
  
      # Loads label file, strips off carriage return
      self.label_lines = [line.rstrip() for line in tf.gfile.GFile("dict.txt")]
  
  def predict(self,sess):
  
      # Feed the image_data as input to the graph and get first prediction
      softmax_tensor = sess.graph.get_tensor_by_name('final_ops/softmax:0')
  
      predictions = sess.run(softmax_tensor, {'Placeholder:0': self.tfimages})
      for prediction in predictions:
          # Sort to show labels of first prediction in order of confidence
          top_k = prediction.argsort()[-len(prediction):][::-1]
  
          for node_id in top_k:
              human_string = self.label_lines[node_id]
              score = prediction[node_id]
              print('%s (score = %.5f)' % (human_string, score))
          return(human_string)
  
  if __name__ == "__main__":
      with tf.Session(graph=tf.Graph()) as sess:
          tf.saved_model.loader.load(sess,[tf.saved_model.tag_constants.SERVING], "model")
          tensorflow_instance=tensorflow(read_from="file")
          tensorflow_instance.predict(sess)
  
      sess=tf.Session(graph=tf.Graph())
      tf.saved_model.loader.load(sess,[tf.saved_model.tag_constants.SERVING], "model")
      tensorflow_instance=tensorflow(read_from="file")
      tensorflow_instance.predict(sess)
  </code></pre>