title: Using binary_crossentropy loss in Keras (Tensorflow backend)
body: <p>In the training example in Keras documentation,</p>
  
  <p><a href="https://keras.io/getting-started/sequential-model-guide/#training" rel="noreferrer">https://keras.io/getting-started/sequential-model-guide/#training</a></p>
  
  <p><strong>binary_crossentropy</strong> is used and <strong>sigmoid</strong> activation is added in the network's last layer, but is it necessary that add <strong>sigmoid</strong> in the last layer? As I found in the source code:</p>
  
  <pre><code>def binary_crossentropy(output, target, from_logits=False):
    """Binary crossentropy between an output tensor and a target tensor.
    Arguments:
        output: A tensor.
        target: A tensor with the same shape as `output`.
        from_logits: Whether `output` is expected to be a logits tensor.
            By default, we consider that `output`
            encodes a probability distribution.
    Returns:
        A tensor.
    """
    # Note: nn.softmax_cross_entropy_with_logits
    # expects logits, Keras expects probabilities.
    if not from_logits:
      # transform back to logits
      epsilon = _to_tensor(_EPSILON, output.dtype.base_dtype)
      output = clip_ops.clip_by_value(output, epsilon, 1 - epsilon)
      output = math_ops.log(output / (1 - output))
    return nn.sigmoid_cross_entropy_with_logits(labels=target, logits=output)
  </code></pre>
  
  <p>Keras invokes <strong>sigmoid_cross_entropy_with_logits</strong> in Tensorflow, but in <strong>sigmoid_cross_entropy_with_logits</strong> function, <strong>sigmoid(logits)</strong> is calculated again.</p>
  
  <p><a href="https://www.tensorflow.org/versions/master/api_docs/python/tf/nn/sigmoid_cross_entropy_with_logits" rel="noreferrer">https://www.tensorflow.org/versions/master/api_docs/python/tf/nn/sigmoid_cross_entropy_with_logits</a></p>
  
  <p>So I don't think it makes sense that add a <strong>sigmoid</strong> at last, but seemingly all the binary/multi-label classification examples and tutorials in Keras I found online added <strong>sigmoid</strong> at last. Besides I don't understand what is the meaning of</p>
  
  <pre><code># Note: nn.softmax_cross_entropy_with_logits
  # expects logits, Keras expects probabilities.
  </code></pre>
  
  <p>Why Keras expects probabilities? Doesn't it use the <strong>nn.softmax_cross_entropy_with_logits</strong> function? Does it make sense?</p>
  
  <p>Thanks.</p>