title: How to convert multi-gpu weights to single GPU model?
body: <p>While loading weights from a multi-gpu trained model into CPU or single GPU model, I get errors.</p>
  
  <p>I used DenseNet121 with imagenet weights, add separate dense layers on top of it due to different activation function properties of the output, and concatenated them in the output layer</p>
  
  <pre class="lang-py prettyprint-override"><code>
  base_model = DenseNet121(include_top=False,weights="imagenet", input_shape = IMG_SHAPE)
  
  x = base_model.output
  x = GlobalAveragePooling2D()(x)
  
  class1 = Dense(512,activation="relu")(x)
  class1 = Dense(3,activation="softmax")(class1)
  
  class2 = Dense(512,activation="relu")(x)
  class2 = Dense(4,activation="softmax")(class2 )
  
  class3 = Dense(512,activation="relu")(x)
  class3 = Dense(5,activation="sigmoid")(class3 )
  
  class4 = Dense(512,activation="relu")(x)
  class4 = Dense(6,activation="softmax")(class4 )
  
  class5 = Dense(512,activation="relu")(x)
  class5 = Dense(7,activation="sigmoid")(class5 )
  
  class6 = Dense(512, activation="relu")(x)
  class6 = Dense(8, activation = "softmax")(class6 )
  
  con = concatenate([class1 ,class2 ,class3 ,class4 ,class5 ,class6 ])
  
  model = Model(inputs = base_model.input, outputs = con)
  
  model = multi_gpu_model(model, gpus = 2)
  
  ## After training is complete:
  
  model.save_weights('model.h5')
  
  </code></pre>
  
  <p>Cannot load while using tf.keras in CPU only mode. Apart from that, I'm not sure how one can use multi-gpu models while doing deployment to web or else due to the conversion problem of weights.</p>
  
  <p>I also tried saving weights with:</p>
  
  <pre class="lang-py prettyprint-override"><code>multi_gpu_model = model.layers[-2]
  
  multi_gpu_model.save_weights("single_gpu_model.h5")
  </code></pre>
  
  <p>it works in single GPU or CPU mode but it doesn't cover all layers' weights IMO.</p>
  
  <p>Many thanks,</p>