title: Parallel fitting of multiple Keras Models on single GPU
body: <p>
  I'm trying to fit multiple small Keras models in parallel on a single GPU. Because of reasons i need to get them out of a list and train them one step at a time. Since I was not lucky with the standard multiprocessing module i use pathos. </p>
  
  <p>What I tried to do is something like this:</p>
  
  <pre class="lang-py prettyprint-override"><code>from pathos.multiprocessing import ProcessPool as Pool
  import tensorflow as tf
  import keras.backend as K
  
  def multiprocess_step(self, model):
      K.set_session(sess)
      with sess.graph.as_default():
          model = step(model, sess)
          return model
  
  def step(model, sess):
      K.set_session(sess)
      with sess.graph.as_default():
          model.fit(x=data['X_train'], y=data['y_train'],
                 batch_size=batch_size
                 validation_data=(data['X_test'], data['y_test']), 
                 verbose=verbose,
                 shuffle=True,
                 initial_epoch=self.step_num - 1)
          return model
  
  config = tf.ConfigProto()
  config.gpu_options.allow_growth = True
  config.gpu_options.visible_device_list = "0"
  sess = tf.Session(config=config)
  
  K.set_session(sess)
  with sess.graph.as_default():
      pool = Pool(8).map
      model_list = pool(multiprocess_step, model_list)
  </code></pre>
  
  <p>but whatever I try I keep getting an error claiming that the models dont seem to be on the same graph...</p>
  
  <p><code>ValueError: Tensor("training/RMSprop/Variable:0", shape=(25, 352), dtype=float32_ref) must be from the same graph as Tensor("RMSprop/rho/read:0", shape=(), dtype=float32).</code></p>
  
  <p>The exception originates in the model.fit() row so I must have done something wrong with the assignment of the session graph even though I tried to set that in every possible location?</p>
  
  <p>Does anyone have experience with something similar?</p>