title: Turning off softmax in tensorflow models
body: <p>All I want to do is download one of tensorflow's built in models (via keras), switch the softmax at the output layer off (i.e. replace it with the linear activation function), so that my output features are the activations on the output layer <em>before</em> softmax is applied. </p>
  
  <p>So, I grab VGG16 as a model, and call it base_model</p>
  
  <pre><code>from tensorflow.keras.applications.vgg16 import VGG16
  from tensorflow.keras.applications.vgg16 import preprocess_input
  base_model = VGG16()
  </code></pre>
  
  <p>I have a look at the final layer like this: </p>
  
  <pre><code>base_model.get_layer('predictions').get_config()
  </code></pre>
  
  <p>and get: </p>
  
  <pre><code>{'name': 'predictions',
   'trainable': True,
   'dtype': 'float32',
   'units': 1000,
   'activation': 'softmax',
   'use_bias': True,
   'kernel_initializer': {'class_name': 'GlorotUniform',
    'config': {'seed': None, 'dtype': 'float32'}},
   'bias_initializer': {'class_name': 'Zeros', 'config': {'dtype': 'float32'}},
   'kernel_regularizer': None,
   'bias_regularizer': None,
   'activity_regularizer': None,
   'kernel_constraint': None,
   'bias_constraint': None}
  </code></pre>
  
  <p>Then, I do this to switch activation functions:</p>
  
  <pre><code>base_model.get_layer('predictions').activation=tf.compat.v1.keras.activations.linear
  </code></pre>
  
  <p>and it looks like it works as :</p>
  
  <pre><code>base_model.get_layer('predictions').get_config()
  </code></pre>
  
  <p>gives: </p>
  
  <pre><code>{'name': 'predictions',
   'trainable': True,
   'dtype': 'float32',
   'units': 1000,
   'activation': 'linear',
   'use_bias': True,
   'kernel_initializer': {'class_name': 'GlorotUniform',
    'config': {'seed': None, 'dtype': 'float32'}},
   'bias_initializer': {'class_name': 'Zeros', 'config': {'dtype': 'float32'}},
   'kernel_regularizer': None,
   'bias_regularizer': None,
   'activity_regularizer': None,
   'kernel_constraint': None,
   'bias_constraint': None}.
  </code></pre>
  
  <p>But when I put in a picture, using: </p>
  
  <pre><code>filename = 'test_data/ILSVRC2012_val_00001218.JPEG'
  img = image.load_img(filename, target_size=(224, 224)) # loads image
  x = image.img_to_array(img) # convets to a numpy array
  x = np.expand_dims(x, axis=0) # batches images
  x = preprocess_input(x) # prepare the image for the VGG model
  </code></pre>
  
  <p>and I do a predict on it, to get my features:</p>
  
  <pre><code>features = base_model.predict(x)
  </code></pre>
  
  <p>The feature still sum to 1, i.e. they look like they have been normalised by softmax as </p>
  
  <pre><code>sum(features[0])
  </code></pre>
  
  <p>is 1.0000000321741935, which is the exact same number I got when I did this with the softmax activation function on that layer. </p>
  
  <p>I also tried copying out the config dictionary with 'linear' in it, and using set_config on the output layer.</p>
  
  <p>Turning off softmax seems to be bizarrely hard to do in tensorflow: in caffe, you can just switch activation functions for a pre-trained model by just changing one line in the deploy file, so I really don't understand why this is so difficult in tensorflow. I'm after switching my code from caffe to tensorflow, as I thought that it would be easier to use tf to just grab pre-trained models, but this issue is making me reconsider. </p>
  
  <p>I supposed I could try to rip off the prediction layer and replace it with a brand new one with all the same settings (and put the old weights in), but I am sure there must be way to just edit the prediction layer. </p>
  
  <p>I'm using TensorFlow 1.14.0 at the moment, I'm planning to upgrade to 2.0, but I don't think that using tensorflow 1 is the problem here. </p>
  
  <p>Can anyone explain to me how to turn off softmax please? It should be a simple thing to do and I've spent hours on it and have even joined stack overflow just to get this single issue fixed. </p>
  
  <p>Thanks in advance for any help. </p>