title: What is the difference between a sigmoid followed by the cross entropy and sigmoid_cross_entropy_with_logits in TensorFlow?
body: <p>When trying to get cross-entropy with sigmoid activation function, there is a difference between </p>
  
  <ol>
  <li><code>loss1 = -tf.reduce_sum(p*tf.log(q), 1)</code></li>
  <li><code>loss2 = tf.reduce_sum(tf.nn.sigmoid_cross_entropy_with_logits(labels=p, logits=logit_q),1)</code></li>
  </ol>
  
  <p>But they are the same when with softmax activation function.</p>
  
  <p>Following is the sample code:</p>
  
  
  
  <pre class="lang-python prettyprint-override"><code>import tensorflow as tf
  
  sess2 = tf.InteractiveSession()
  p = tf.placeholder(tf.float32, shape=[None, 5])
  logit_q = tf.placeholder(tf.float32, shape=[None, 5])
  q = tf.nn.sigmoid(logit_q)
  sess.run(tf.global_variables_initializer())
  
  feed_dict = {p: [[0, 0, 0, 1, 0], [1,0,0,0,0]], logit_q: [[0.2, 0.2, 0.2, 0.2, 0.2], [0.3, 0.3, 0.2, 0.1, 0.1]]}
  loss1 = -tf.reduce_sum(p*tf.log(q),1).eval(feed_dict)
  loss2 = tf.reduce_sum(tf.nn.sigmoid_cross_entropy_with_logits(labels=p, logits=logit_q),1).eval(feed_dict)
  
  print(p.eval(feed_dict), "\n", q.eval(feed_dict))
  print("\n",loss1, "\n", loss2)
  </code></pre>