title: Activation functions: Softmax vs Sigmoid
body: <p>I've been trying to build an image classifier with CNN. There are 2300 images in my dataset and two categories: men and women. Here's the model I used:</p>
  <pre><code>early_stopping = EarlyStopping(min_delta = 0.001, patience = 30, restore_best_weights = True)
  model = tf.keras.Sequential()
  
  model.add(tf.keras.layers.Conv2D(256, (3, 3), input_shape=X.shape[1:],  activation = 'relu'))
  
  model.add(tf.keras.layers.MaxPooling2D(pool_size=(2, 2)))
  model.add(tf.keras.layers.BatchNormalization())
  model.add(tf.keras.layers.Conv2D(256, (3, 3), input_shape=X.shape[1:], activation = 'relu'))
  model.add(tf.keras.layers.MaxPooling2D(pool_size=(2, 2)))
  model.add(tf.keras.layers.BatchNormalization())
  model.add(tf.keras.layers.Flatten())  # this converts our 3D feature maps to 1D feature vectors
  
  model.add(tf.keras.layers.Dense(64))
  
  model.add(tf.keras.layers.Dense(1, activation='softmax'))
  
  
  model.compile(loss='binary_crossentropy',
                optimizer='adam',
                metrics=['accuracy'])
  
  h= model.fit(xtrain, ytrain, validation_data=(xval, yval), batch_size=32, epochs=30, callbacks = [early_stopping], verbose = 0)
  </code></pre>
  <p>Accuracy of this model is 0.501897 and loss 7.595693(the model is stuck on these numbers in every epoch) but if I replace Softmax activation with Sigmoid, accuracy is about 0.98 and loss 0.06. Why does such strange thing happen with Softmax? All info I could find was that these two activations are similar and softmax is even better but I couldn't find anything about such abnormality. I'll be glad if someone could explain what the problem is.</p>