title: Choosing from different cost function and activation function of a neural network
body: <p>Recently I started toying with neural networks. I was trying to implement an <code>AND</code> gate with Tensorflow. I am having trouble understanding when to use different cost and activation functions. This is a basic neural network with only input and output layers, no hidden layers.</p>
  
  <p>First I tried to implement it in this way. As you can see this is a poor implementation, but I think it gets the job done, at least in some way. So, I tried only the real outputs, no one hot true outputs. For activation functions, I used a sigmoid function and for cost function I used squared error cost function (I think its called that, correct me if I'm wrong). </p>
  
  <p>I've tried using ReLU and Softmax as activation functions (with the same cost function) and it doesn't work. I figured out why they don't work. I also tried the sigmoid function with Cross Entropy cost function, it also doesn't work.</p>
  
  <pre><code>import tensorflow as tf
  import numpy
  
  train_X = numpy.asarray([[0,0],[0,1],[1,0],[1,1]])
  train_Y = numpy.asarray([[0],[0],[0],[1]])
  
  x = tf.placeholder("float",[None, 2])
  y = tf.placeholder("float",[None, 1])
  
  W = tf.Variable(tf.zeros([2, 1]))
  b = tf.Variable(tf.zeros([1, 1]))
  
  activation = tf.nn.sigmoid(tf.matmul(x, W)+b)
  cost = tf.reduce_sum(tf.square(activation - y))/4
  optimizer = tf.train.GradientDescentOptimizer(.1).minimize(cost)
  
  init = tf.initialize_all_variables()
  
  with tf.Session() as sess:
      sess.run(init)
      for i in range(5000):
          train_data = sess.run(optimizer, feed_dict={x: train_X, y: train_Y})
  
      result = sess.run(activation, feed_dict={x:train_X})
      print(result)
  </code></pre>
  
  <p>after 5000 iterations:</p>
  
  <pre><code>[[ 0.0031316 ]
  [ 0.12012422]
  [ 0.12012422]
  [ 0.85576665]]
  </code></pre>
  
  <p><strong>Question 1</strong> - Is there any other activation function and cost function, that can work(learn) for the above network, without changing the parameters(meaning without changing W, x, b). </p>
  
  <p><strong>Question 2</strong> - I read from a StackOverflow post <a href="https://stackoverflow.com/questions/20850895/neural-networks-activation-function">here</a>:</p>
  
  <blockquote>
    <p>[Activation Function] selection depends on the problem.</p>
  </blockquote>
  
  <p>So there are no cost functions that can be used anywhere? I mean there is no <em>standard</em> cost function that can be used on any neural network. Right? Please correct me on this.</p>
  
  <p><br/></p>
  
  <p>I also implemented the <code>AND</code> gate with a different approach, with the output as one-hot true. As you can see the <code>train_Y</code> <code>[1,0]</code> means that the 0th index is 1, so the answer is 0. I hope you get it. </p>
  
  <p>Here I have used a softmax activation function, with cross entropy as cost function. Sigmoid function as activation function fails miserably. </p>
  
  <pre><code>import tensorflow as tf
  import numpy
  
  train_X = numpy.asarray([[0,0],[0,1],[1,0],[1,1]])
  train_Y = numpy.asarray([[1,0],[1,0],[1,0],[0,1]])
  
  x = tf.placeholder("float",[None, 2])
  y = tf.placeholder("float",[None, 2])
  
  W = tf.Variable(tf.zeros([2, 2]))
  b = tf.Variable(tf.zeros([2]))
  
  activation = tf.nn.softmax(tf.matmul(x, W)+b)
  
  cost = -tf.reduce_sum(y*tf.log(activation))
  
  optimizer = tf.train.GradientDescentOptimizer(0.5).minimize(cost)
  
  init = tf.initialize_all_variables()
  
  with tf.Session() as sess:
      sess.run(init)
      for i in range(5000):
          train_data = sess.run(optimizer, feed_dict={x: train_X, y: train_Y})
  
      result = sess.run(activation, feed_dict={x:train_X})
      print(result)
  </code></pre>
  
  <p>after 5000 iteration</p>
  
  <pre><code>[[  1.00000000e+00   1.41971401e-09]
   [  9.98996437e-01   1.00352429e-03]
   [  9.98996437e-01   1.00352429e-03]
   [  1.40495342e-03   9.98595059e-01]]
  </code></pre>
  
  <p><strong>Question 3</strong> So in this case what cost function and activation function can I use? How do I understand what type of cost and activation functions I should use? Is there a standard way or rule, or just experience only? Should I have to try every cost and activation function in a brute force manner? I found an answer <a href="https://stackoverflow.com/questions/20368015/activation-function-choice-for-neural-network">here</a>. But I am hoping for a more elaborate explanation. </p>
  
  <p><strong>Question 4</strong> I have noticed that it takes many iterations to converge to a near accurate prediction. I think the convergance rate depends on the learning rate (using too large of will miss the solution) and the cost function (correct me if I'm wrong). So, is there any optimal way (meaning the fastest) or cost function for converging to a correct solution?</p>