title: Tensorflow: Error when loading model trained using tf.layers API
body: <p>I have asked this before but haven't been able to find help. I have a model trained using Tensorflow which I am attempting to load. I can load models I write using only tf.nn.xxxx functions just fine, but anything I write using tf.layers I cannot load. I have spent months trying to find soltuions to this, but no one seems to have an identical problem online which leads me to believe I'm doing something very subtly stupid. I'm immensely grateful for any help at this stage</p>
  
  <p>When I try to run data through the tf.layers model in inference time  I am given the following error message:</p>
  
  <pre><code>FailedPreconditionError (see above for traceback): Attempting to use uninitialized value Level3Decoding/conv3/batch_normalization/beta_1
       [[Node: Level3Decoding/conv3/batch_normalization/beta_1/read = Identity[T=DT_FLOAT, _class=["loc:@Level3Decoding/conv3/batch_normalization/cond/FusedBatchNorm_3/Switch_2"], _device="/job:localhost/replica:0/task:0/device:GPU:0"](Level3Decoding/conv3/batch_normalization/beta_1)]]
       [[Node: Level1Decoding/conv2d/BiasAdd_1/_811 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_1214_Level1Decoding/conv2d/BiasAdd_1", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]
  </code></pre>
  
  <p>I am training a network for segmentation, which is defined thusly:</p>
  
  <pre><code>def uNet2D(x, REGULARIZER, KERNEL_SIZE, is_training):
  regularizer = tf.contrib.layers.l2_regularizer(scale=REGULARIZER)
  
  #L1 encode
  with tf.variable_scope('Level1Encoding'):
      with tf.variable_scope('conv1'):
          conv1=tf.layers.conv2d(x,64,(KERNEL_SIZE, KERNEL_SIZE), kernel_regularizer=regularizer, padding='same', name='conv1')
          conv1  = tf.layers.batch_normalization(
          inputs=conv1,
          axis=-1,
          momentum=0.999,
          epsilon=0.001,
          center=True,
          scale=True,
          training = is_training)
          conv1 = tf.nn.relu(conv1)
  
      with tf.variable_scope('conv2'):
          conv2=tf.layers.conv2d(conv1,64,(KERNEL_SIZE, KERNEL_SIZE), kernel_regularizer=regularizer, padding='same')
          conv2  = tf.layers.batch_normalization(
          inputs=conv2,
          axis=-1,
          momentum=0.999,
          epsilon=0.001,
          center=True,
          scale=True,
          training = is_training)
          conv2 = tf.nn.relu(conv2)
      with tf.variable_scope('conv3'):
          conv3=tf.layers.conv2d(conv2,64,(KERNEL_SIZE, KERNEL_SIZE), kernel_regularizer=regularizer, padding='same')
          conv3  = tf.layers.batch_normalization(
          inputs=conv3,
          axis=-1,
          momentum=0.999,
          epsilon=0.001,
          center=True,
          scale=True,
          training = is_training)
          conv3 = tf.nn.relu(conv3)
  
      conv3mp=tf.layers.max_pooling2d(conv3,2,2,padding='same')
  
  with tf.variable_scope('Level2Encoding'):
      with tf.variable_scope('conv1'):
          conv4=tf.layers.conv2d(conv3mp,128,(KERNEL_SIZE, KERNEL_SIZE), kernel_regularizer=regularizer, padding='same')
          conv4  = tf.layers.batch_normalization(
          inputs=conv4,
          axis=-1,
          momentum=0.999,
          epsilon=0.001,
          center=True,
          scale=True,
          training = is_training)
          conv4 = tf.nn.relu(conv4)
  
      with tf.variable_scope('conv2'):
          conv5=tf.layers.conv2d(conv4,128,(KERNEL_SIZE, KERNEL_SIZE), kernel_regularizer=regularizer, padding='same')
          conv5  = tf.layers.batch_normalization(
          inputs=conv5,
          axis=-1,
          momentum=0.999,
          epsilon=0.001,
          center=True,
          scale=True,
          training = is_training)
          conv5 = tf.nn.relu(conv5)
  
      with tf.variable_scope('conv3'):
          conv6=tf.layers.conv2d(conv5,128,(KERNEL_SIZE, KERNEL_SIZE), kernel_regularizer=regularizer, padding='same')
          conv6  = tf.layers.batch_normalization(
          inputs=conv6,
          axis=-1,
          momentum=0.999,
          epsilon=0.001,
          center=True,
          scale=True,
          training = is_training)
          conv6 = tf.nn.relu(conv6)
  
      conv6mp=tf.layers.max_pooling2d(conv6,2,2,padding='same')
  
  with tf.variable_scope('Level3Encoding'):
      with tf.variable_scope('conv1'):
          conv7=tf.layers.conv2d(conv6mp,256,(KERNEL_SIZE, KERNEL_SIZE), kernel_regularizer=regularizer, padding='same')
          conv7  = tf.layers.batch_normalization(
          inputs=conv7,
          axis=-1,
          momentum=0.999,
          epsilon=0.001,
          center=True,
          scale=True,
          training = is_training)
          conv7 = tf.nn.relu(conv7)
  
      with tf.variable_scope('conv2'):    
          conv8=tf.layers.conv2d(conv7,256,(KERNEL_SIZE, KERNEL_SIZE), kernel_regularizer=regularizer, padding='same')
          conv8  = tf.layers.batch_normalization(
          inputs=conv8,
          axis=-1,
          momentum=0.999,
          epsilon=0.001,
          center=True,
          scale=True,
          training = is_training)
          conv8 = tf.nn.relu(conv8)
  
      with tf.variable_scope('conv3'):
          conv9=tf.layers.conv2d(conv8,256,(KERNEL_SIZE, KERNEL_SIZE), kernel_regularizer=regularizer, padding='same')
          conv9  = tf.layers.batch_normalization(
          inputs=conv9,
          axis=-1,
          momentum=0.999,
          epsilon=0.001,
          center=True,
          scale=True,
          training = is_training)
          conv9 = tf.nn.relu(conv9)
  
      conv9mp=tf.layers.max_pooling2d(conv9,2,2,padding='same')
  
  with tf.variable_scope('Level4Encoding'):
      with tf.variable_scope('conv1'):
          conv10=tf.layers.conv2d(conv9mp,512,(KERNEL_SIZE, KERNEL_SIZE), kernel_regularizer=regularizer, padding='same')
          conv10  = tf.layers.batch_normalization(
          inputs=conv10,
          axis=-1,
          momentum=0.999,
          epsilon=0.001,
          center=True,
          scale=True,
          training = is_training)
          conv10 = tf.nn.relu(conv10)
  
      with tf.variable_scope('conv2'):
          conv11=tf.layers.conv2d(conv10,512,(KERNEL_SIZE, KERNEL_SIZE), kernel_regularizer=regularizer, padding='same')
          conv11  = tf.layers.batch_normalization(
          inputs=conv11,
          axis=-1,
          momentum=0.999,
          epsilon=0.001,
          center=True,
          scale=True,
          training = is_training)
          conv11 = tf.nn.relu(conv11)
  
      with tf.variable_scope('conv3'):
          conv12=tf.layers.conv2d(conv11,512,(KERNEL_SIZE, KERNEL_SIZE), kernel_regularizer=regularizer, padding='same')
          conv12  = tf.layers.batch_normalization(
          inputs=conv12,
          axis=-1,
          momentum=0.999,
          epsilon=0.001,
          center=True,
          scale=True,
          training = is_training)
          conv12 = tf.nn.relu(conv12)
  
      conv12mp=tf.layers.max_pooling2d(conv12,2,2,padding='same')
  
  
  with tf.variable_scope('Level5'):
      with tf.variable_scope('conv1'):
          conv13=tf.layers.conv2d(conv12mp,1024,(KERNEL_SIZE, KERNEL_SIZE), kernel_regularizer=regularizer, padding='same')
          conv13  = tf.layers.batch_normalization(
          inputs=conv13,
          axis=-1,
          momentum=0.999,
          epsilon=0.001,
          center=True,
          scale=True,
          training = is_training)
          conv13 = tf.nn.relu(conv13)
  
      with tf.variable_scope('conv2'):
          conv14=tf.layers.conv2d(conv13,1024,(KERNEL_SIZE, KERNEL_SIZE), kernel_regularizer=regularizer, padding='same')
          conv14  = tf.layers.batch_normalization(
          inputs=conv14,
          axis=-1,
          momentum=0.999,
          epsilon=0.001,
          center=True,
          scale=True,
          training = is_training)
          conv14 = tf.nn.relu(conv14)
  
      with tf.variable_scope('conv3'):
          conv15=tf.layers.conv2d(conv14,1024,(KERNEL_SIZE, KERNEL_SIZE), kernel_regularizer=regularizer, padding='same')
          conv15  = tf.layers.batch_normalization(
          inputs=conv15,
          axis=-1,
          momentum=0.999,
          epsilon=0.001,
          center=True,
          scale=True,
          training = is_training)
          conv15 = tf.nn.relu(conv15)
  
      conv15=tf.layers.conv2d_transpose(conv15,512,(KERNEL_SIZE, KERNEL_SIZE), kernel_regularizer=regularizer, strides=(2,2),padding='same')
  
  with tf.variable_scope('Level4Decoding'):
      inp     = tf.concat([conv12,conv15],3)
      with tf.variable_scope('conv1'):
          conv16  = tf.layers.conv2d(inp,256,(KERNEL_SIZE, KERNEL_SIZE), kernel_regularizer=regularizer, padding='same')
          conv16  = tf.layers.batch_normalization(
                  inputs=conv16,
                  axis=-1,
                  momentum=0.999,
                  epsilon=0.001,
                  center=True,
                  scale=True,
                  training = is_training)
          conv16 = tf.nn.relu(conv16)
  
      with tf.variable_scope('conv2'):
          conv17=tf.layers.conv2d(conv16,256,(KERNEL_SIZE, KERNEL_SIZE), kernel_regularizer=regularizer, padding='same')
          conv17  = tf.layers.batch_normalization(
                  inputs=conv17,
                  axis=-1,
                  momentum=0.999,
                  epsilon=0.001,
                  center=True,
                  scale=True,
                  training = is_training)
          conv17 = tf.nn.relu(conv17)
  
      with tf.variable_scope('conv3'):
          conv18=tf.layers.conv2d(conv17,256,(KERNEL_SIZE, KERNEL_SIZE), kernel_regularizer=regularizer, padding='same')
          conv18  = tf.layers.batch_normalization(
                  inputs=conv18,
                  axis=-1,
                  momentum=0.999,
                  epsilon=0.001,
                  center=True,
                  scale=True,
                  training = is_training)
          conv18 = tf.nn.relu(conv18)
  
      conv18=tf.layers.conv2d_transpose(conv18,256,(KERNEL_SIZE, KERNEL_SIZE),strides=(2,2), kernel_regularizer=regularizer, padding='same')
  
  with tf.variable_scope('Level3Decoding'):
      inp     = tf.concat([conv9,conv18],3)
      with tf.variable_scope('conv1'):
          conv19  = tf.layers.conv2d(inp,128,(KERNEL_SIZE, KERNEL_SIZE), kernel_regularizer=regularizer, padding='same')
          conv19  = tf.layers.batch_normalization(
                  inputs=conv19,
                  axis=-1,
                  momentum=0.999,
                  epsilon=0.001,
                  center=True,
                  scale=True,
                  training = is_training)
          conv19 = tf.nn.relu(conv19)
  
      with tf.variable_scope('conv2'):
          conv20=tf.layers.conv2d(conv19,128,(KERNEL_SIZE, KERNEL_SIZE), kernel_regularizer=regularizer, padding='same')
          conv20  = tf.layers.batch_normalization(
                  inputs=conv20,
                  axis=-1,
                  momentum=0.999,
                  epsilon=0.001,
                  center=True,
                  scale=True,
                  training = is_training)
          conv20 = tf.nn.relu(conv20)
  
      with tf.variable_scope('conv3'):
          conv21=tf.layers.conv2d(conv20,128,(KERNEL_SIZE, KERNEL_SIZE), kernel_regularizer=regularizer, padding='same')
          conv21  = tf.layers.batch_normalization(
                  inputs=conv21,
                  axis=-1,
                  momentum=0.999,
                  epsilon=0.001,
                  center=True,
                  scale=True,
                  training = is_training)
          conv21 = tf.nn.relu(conv21)
  
      conv21=tf.layers.conv2d_transpose(conv21,128,(KERNEL_SIZE, KERNEL_SIZE), kernel_regularizer=regularizer, strides=(2,2),padding='same')
  
  with tf.variable_scope('Level2Decoding'):
      inp     = tf.concat([conv6,conv21],3)
      with tf.variable_scope('conv1'):
          conv22  = tf.layers.conv2d(inp,64,(KERNEL_SIZE, KERNEL_SIZE), kernel_regularizer=regularizer, padding='same')
          conv22  = tf.layers.batch_normalization(
                  inputs=conv22,
                  axis=-1,
                  momentum=0.999,
                  epsilon=0.001,
                  center=True,
                  scale=True,
                  training = is_training)
          conv22 = tf.nn.relu(conv22)
  
      with tf.variable_scope('conv2'):
          conv23  = tf.layers.conv2d(conv22,128,(KERNEL_SIZE, KERNEL_SIZE), kernel_regularizer=regularizer, padding='same')
          conv23  = tf.layers.batch_normalization(
                  inputs=conv23,
                  axis=-1,
                  momentum=0.999,
                  epsilon=0.001,
                  center=True,
                  scale=True,
                  training = is_training)
          conv23 = tf.nn.relu(conv23)
  
      with tf.variable_scope('conv3'):
          conv24=tf.layers.conv2d(conv23,128,(KERNEL_SIZE, KERNEL_SIZE), kernel_regularizer=regularizer, padding='same')
          conv24  = tf.layers.batch_normalization(
                  inputs=conv24,
                  axis=-1,
                  momentum=0.999,
                  epsilon=0.001,
                  center=True,
                  scale=True,
                  training = is_training)
          conv24 = tf.nn.relu(conv24)
  
      conv24=tf.layers.conv2d_transpose(conv24,64,(KERNEL_SIZE, KERNEL_SIZE), kernel_regularizer=regularizer, strides=(2,2),padding='same')
  
  with tf.variable_scope('Level1Decoding'):
      inp     = tf.concat([conv3,conv24],3)
      with tf.variable_scope('conv1'):
          conv25  = tf.layers.conv2d(inp,64,(KERNEL_SIZE, KERNEL_SIZE), kernel_regularizer=regularizer, padding='same')
          conv25  = tf.layers.batch_normalization(
                  inputs=conv25,
                  axis=-1,
                  momentum=0.999,
                  epsilon=0.001,
                  center=True,
                  scale=True,
                  training = is_training)
          conv25 = tf.nn.relu(conv25)
  
      with tf.variable_scope('conv2'):
          conv26  = tf.layers.conv2d(conv25,128,(KERNEL_SIZE, KERNEL_SIZE), kernel_regularizer=regularizer, padding='same')
          conv26  = tf.layers.batch_normalization(
                  inputs=conv25,
                  axis=-1,
                  momentum=0.999,
                  epsilon=0.001,
                  center=True,
                  scale=True,
                  training = is_training)
          conv26 = tf.nn.relu(conv26)
  
      with tf.variable_scope('conv3'):
          conv27  = tf.layers.conv2d(conv26,128,(KERNEL_SIZE, KERNEL_SIZE), kernel_regularizer=regularizer, padding='same')
          conv27  = tf.layers.batch_normalization(
                  inputs=conv26,
                  axis=-1,
                  momentum=0.999,
                  epsilon=0.001,
                  center=True,
                  scale=True,
                  training = is_training)
          conv27=tf.nn.relu(conv27)
  
      convOUT = tf.layers.conv2d(conv27,1,(1,1),  kernel_regularizer=regularizer, padding='same')
      return convOUT
  </code></pre>
  
  <p>I am training it with following script:</p>
  
  <pre><code>    os.chdir(launch)
  
  timestr = time.strftime("%Y%m%d-%H%M%S")
  outDir  = './TumorOUT_no_core/' + timestr
  os.mkdir(outDir)
  
  graphDir = './graphs/' + timestr
  os.mkdir(graphDir)
  os.mkdir(graphDir + '/training/')
  os.mkdir(graphDir + '/testing/')
  
  
  with open(outDir + '/hyperparams.txt', 'w+') as f: 
      for key, value in hypers.items():
          f.write('%s:%s\n' % (key, value))
  
  
  X = tf.placeholder(tf.float32, shape=[None, None, None, NUM_CHANNELS],   name='X')  #input
  Y = tf.placeholder(tf.float32, shape = [None, None, None, 1], name='Y')  #'labels'
  is_training = tf.placeholder(tf.bool, name='is_training')
  
  def run_model():
  
          valCosts=[]
          GLOBAL_STEP = 0
          minLoss=10000000000
          pred = uNet2D(X, BETA, KERNEL_SIZE, is_training)
          cost = tf.reduce_sum(tf.nn.sigmoid_cross_entropy_with_logits(labels=tf.reshape(Y,[-1]),logits=tf.reshape(pred,[-1])))
  
          #with tf.variable_scope('prediction') as scope:
          #    t_pred = uNet2D(X, BETA, KERNEL_SIZE)  
          #    t_cost = tf.reduce_sum(tf.nn.sigmoid_cross_entropy_with_logits(labels=tf.reshape(Y,[-1]),logits=tf.reshape(t_pred,[-1])))
          #    scope.reuse_variables()
          #    v_pred = uNet2D(X, BETA, KERNEL_SIZE)  
          #    v_cost = tf.reduce_sum(tf.nn.sigmoid_cross_entropy_with_logits(labels=tf.reshape(Y,[-1]),logits=tf.reshape(v_pred,[-1])))
  
          update_ops = tf.get_collection(tf.GraphKeys.UPDATE_OPS)
          with tf.control_dependencies(update_ops):
          # this with statement updates the BN parameters if BN is being used
              optimizer = tf.train.AdamOptimizer(learning_rate=LR).minimize(cost)
  
          with tf.name_scope("training"):
              tf.summary.scalar("training_cost", cost, collections=['training'])
              for var in tf.trainable_variables():
                  tf.summary.histogram(var.op.name , var, collections=['training'])
  
          with tf.name_scope("validation"):
              tf.summary.scalar("validation_cost", cost, collections=['validation'])
              #tf.summary.image("VALIDATION_X",X, collections=['validation'])
              #tf.summary.image("VALIDATION_Y",Y, collections=['validation'])
              #tf.summary.image("VALIDATION_PRED", v_pred, collections=['validation'])
  
  
          saver = tf.train.Saver()
  
          with tf.Session() as sess:
              train_merge      = tf.summary.merge_all(key='training')
              validation_merge = tf.summary.merge_all(key='validation')
              train_writer = tf.summary.FileWriter( graphDir + '/training/', sess.graph)
              validation_writer = tf.summary.FileWriter( graphDir + '/testing/', sess.graph)
              print('Beginning Session!')
              sess.run(tf.global_variables_initializer())
              print('Running Model!')
  
              while GLOBAL_STEP &lt;= MAX_ITER:
  
                  if GLOBAL_STEP % NUM_STEPS != 0:
                      x,y=training.drawBatch(BATCH_SIZE)
                      y=np.expand_dims(y,-1)
                      y[y&gt;0]=1
                      flip = random.uniform(0,1)
                      if flip&gt;0.5:
                          x = np.flip(x,2)
                          y = np.flip(y,2)
                      _, acc, c = sess.run([optimizer, train_merge, cost], feed_dict = {X: x, Y: y, is_training: 1})
                      train_writer.add_summary(acc, GLOBAL_STEP)
                      #train_writer.add_summary(summary, GLOBAL_STEP)
                  else:
                      x,y=validation.drawBatch(BATCH_SIZE)
                      y=np.expand_dims(y,-1)
                      y[y&gt;0]=1
                      acc, c = sess.run([validation_merge,cost], feed_dict = {X: x, Y: y, is_training: 0})
                      validation_writer.add_summary(acc, GLOBAL_STEP)
                      #validation_writer.add_summary(summary, GLOBAL_STEP)
                      #save_path=saver.save(sess, outDir + '/model')
                      print(c)
                      if c &lt; minLoss:
                          save_path=saver.save(sess, outDir + '/model')
                          minLoss=c
                      valCosts.append(c)
  
                  GLOBAL_STEP+=1
              g= open(graphDir+'/val.pickle', 'w+')   
              pickle.dump([valCosts], g)
  
  run_model()
  </code></pre>
  
  <p>And I try to load it with the following code, it loads successfully but it seems that I cannot properly initialize my variables:</p>
  
  <pre><code>    launch=os.getcwd()
  
  tf.reset_default_graph()
  os.environ["CUDA_VISIBLE_DEVICES"]="0"
  
  launch=os.getcwd()
  testDir=launch + '/in_data/no_core/Validation/'
  
  os.chdir(testDir)
  testList  = glob.glob('*mat')
  
  
  os.chdir(launch)
  sess = tf.Session()
  new_saver = tf.train.import_meta_graph(launch + '/TumorOUT_no_core/20180829-142646/model.meta')
  new_saver.restore(sess,tf.train.latest_checkpoint(launch + '/TumorOUT_no_core/20180829-142646/'))
  
  
  
  X = tf.placeholder(tf.float32, shape=[None, None, None, 2],   name='X')  #input
  Y = tf.placeholder(tf.float32, shape = [None, None, None, 1], name='Y')  #'labels'
  is_training = tf.placeholder(tf.bool, name='is_training')
  sess.run(tf.global_variables_initializer())
  sess.run(tf.local_variables_initializer())
  
  os.chdir(testDir)
  
  
  
  pred = uNet2D(X, .1, 3,is_training)
  print("Loaded Weights")
  
  
  for i in testList:
      print(i)
      data=loadmat(i)
      mri=data['data']
      mri=mri[:,:,:,0:2]
      outt=np.zeros((mri.shape[0],mri.shape[1],mri.shape[2]))
      mri=(mri-np.mean(mri))/np.var(mri)
      mask=mri[:,:,:,0]&gt;15
      mri=np.expand_dims(mri,0)
      roi=data['roi_mat']
      for z in range(mri.shape[3]):
          b1=sess.run(pred,feed_dict={X: mri[:,:,:,z,:],is_training:False})
          b=b1[0,:,:,0]
          b=sess.run(tf.nn.sigmoid(b))
          r=roi[:,:,z]
          r=r.astype(dtype=np.float32)
          outt[:,:,z]=b
      out={}
      out['out']= outt
      out['roi']= roi
      savemat('./matlab/'+i,out)
  
      print('done :)')
  </code></pre>