title: Training MNIST dataset with Sigmoid Activation function
body: <p>According to <a href="https://iamtrask.github.io/2015/07/12/basic-python-network/" rel="nofollow noreferrer">this tutorial</a>, with <code>Python</code> and <code>Numpy</code>, I want to train <code>MNIST</code> dataset to a neural network that can recognize handwritten digits. I understand the logic but i have a problem right now. </p>
  
  <p>In this tutorial test case was <code>AND</code> logical gate and because of small amount of data, it works fine. But I'm using <code>MNIST</code> database and each image has <code>28*28</code> dimension,And when I convert each of them to a vector, I have <code>N*784</code> matrix. And if I have a <code>784*1</code> matrix as <code>weight matrix</code>, When I multiply it with the input matrix, The resulting numbers is going to be very small or very large numbers(Negative or Positive) and because i use <code>Sigmoid</code> activation function, all of my data divided in to two section, 1 and 0 at first learning cycle, But i need small numbers that converging slowly.</p>
  
  <p>for example i get these numbers after multiplication: <code>-569.87541502</code> , <code>218.62477264</code> and the first one is 0 in <code>Sigmoid</code> activation function and the second is 1, And there is no room for training and converging. All of this because of large amount of data that summation of them resulting to this large or very small numbers.</p>
  
  <p>I use this trick for generating very small weights than the original tutorial but i get the same results(I was thinking because these are small numbers, summation of them can't be very large but i get the same results):</p>
  
  <pre><code>syn0 = (2*np.random.random((784,10))-1)*(0.00000005-0.00000001) + 0.00000001
  </code></pre>
  
  <p>I don't know how i can overcome to this.</p>