title: neural network cost doesn't decrease
body: <p>When i run my neural network the cost always stays around its initial value. sometime it increases and sometimes it decreases but never by much.</p>
  
  <p>I tried some different seeds but this didn't work. I also tried with different input but this also didn't work. </p>
  
  <p>I looked over the math for the training but I couldn't find anything wrong with it.</p>
  
  <pre><code>class NeuralNet():
      def __init__(self):
          np.random.seed(1)
  
          self.w0 = 2*np.random.random((3, 4))-1
          self.w1 = 2*np.random.random((4, 1))-1
  
      def sigmoid(self, x):
          return 1/(1+np.exp(-x))
  
      def sigmoid_d(self, x):
          x = self.sigmoid(x)
          return x * (1-x)
  
      def train(self, training_length, x, y, print_cost_cycle=1000):
          for i in range(training_length):
              l0 = x
              l1 = self.sigmoid(np.dot(l0, self.w0))
              l2 = self.sigmoid(np.dot(l1, self.w1))
  
              l2_error = (l2 - y)**2
              l2_d = 2*(y - l2) * self.sigmoid_d(l2)
              l1_cost = np.dot(l2_d, self.w1.T)
              l1_d = l1_cost * self.sigmoid_d(l1)
  
              self.w0 += np.dot(l0.T, l1_d)
              self.w1 += np.dot(l1.T, l2_d)
  
              if i % print_cost_cycle == 0:
                  print(f"cost = {np.mean(l2_error)}")
  
  
  x_train = np.array([
      [0, 0, 1],
      [0, 1, 1],
      [1, 0, 1],
      [1, 1, 1]
  ])
  y_train = np.array([
      [0],
      [1],
      [1],
      [0]
  ])
  
  neural_net = NeuralNet()
  neural_net.train(10000, x_train, y_train)
  </code></pre>
  
  <p>the output:</p>
  
  <pre><code>cost = 0.2475932677010128
  cost = 0.24844153430705612
  cost = 0.21300873346681945
  cost = 0.21497568086911634
  cost = 0.22868625594402198
  cost = 0.23076981080756687
  cost = 0.23242035817817885
  cost = 0.23560061594143974
  cost = 0.23578477135747
  cost = 0.2356102896524483
  </code></pre>