title: How to create a simple Gradient Descent algorithm
body: <p>I'm studying simple machine learning algorithms, beginning with a simple gradient descent, but I've got some trouble trying to implement it in python. </p>
  
  <p>Here is the example I'm trying to reproduce, I've got data about houses with the (living area (in feet2), and number of bedrooms) with the resulting price :</p>
  
  <p>Living area (feet2) : 2104</p>
  
  <p>#bedrooms : 3</p>
  
  <p>Price (1000$s) : 400</p>
  
  <p>I'm trying to do a simple regression using the gradient descent method, but my algorithm won't work...
  The form of the algorithm is not using vectors on purpose (I'm trying to understand it step by step). </p>
  
  <pre><code>i = 1
  import sys
  derror=sys.maxint
  error = 0
  step = 0.0001
  dthresh = 0.1
  import random
  
  theta1 = random.random()
  theta2 = random.random()
  theta0 = random.random()
  while derror&gt;dthresh:
      diff = 400 - theta0 - 2104 * theta1 - 3 * theta2
      theta0 = theta0 + step * diff * 1
      theta1 = theta1 + step * diff * 2104
      theta2 = theta2 + step * diff * 3
      hserror = diff**2/2
      derror = abs(error - hserror)
      error = hserror
      print 'iteration : %d, error : %s' % (i, error)
      i+=1
  </code></pre>
  
  <p>I understand the math, I'm constructing a predicting function 
  <a href="https://i.stack.imgur.com/Fg3N0.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/Fg3N0.png" alt="$$h_{\theta}(x) = \theta_0 + \theta_1 x_1 + \theta_2 x_2$$"></a>
  
  with <a href="https://i.stack.imgur.com/3Wh5R.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/3Wh5R.png" alt="$x_1$"></a>
   and <a href="https://i.stack.imgur.com/bH9vh.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/bH9vh.png" alt="$x_2$"></a>
   being the variables (living area, number of bedrooms) and <a href="https://i.stack.imgur.com/Am0zP.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/Am0zP.png" alt="$h_{\theta}(x)$"></a>
   the estimated price.</p>
  
  <p>I'm using the cost function (<a href="https://i.stack.imgur.com/045qB.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/045qB.png" alt="$hserror$"></a>
  ) (for one point) :
  <a href="https://i.stack.imgur.com/peDer.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/peDer.png" alt="$$hserror = \frac{1}{2} (h_{\theta}(x) - y)^2$$"></a>
  
  This is a usual problem, but I'm more of a software engineer and I'm learning one step at a time, can you tell me what's wrong ?</p>
  
  <p>I got it working with this code :</p>
  
  <pre><code>data = {(2104, 3) : 400, (1600,3) : 330, (2400, 3) : 369, (1416, 2) : 232, (3000, 4) : 540}
  for x in range(10):
      i = 1
      import sys
      derror=sys.maxint
      error = 0
      step = 0.00000001
      dthresh = 0.0000000001
      import random
  
      theta1 = random.random()*100
      theta2 = random.random()*100
      theta0 = random.random()*100
      while derror&gt;dthresh:
          diff = 400 - (theta0 + 2104 * theta1 + 3 * theta2)
          theta0 = theta0 + step * diff * 1
          theta1 = theta1 + step * diff * 2104
          theta2 = theta2 + step * diff * 3
          hserror = diff**2/2
          derror = abs(error - hserror)
          error = hserror
          #print 'iteration : %d, error : %s, derror : %s' % (i, error, derror)
          i+=1
      print ' theta0 : %f, theta1 : %f, theta2 : %f' % (theta0, theta1, theta2)
      print ' done : %f' %(theta0 + 2104 * theta1 + 3*theta2)
  </code></pre>
  
  <p>which ends up with answers like this :</p>
  
  <pre><code> theta0 : 48.412337, theta1 : 0.094492, theta2 : 50.925579
   done : 400.000043
   theta0 : 0.574007, theta1 : 0.185363, theta2 : 3.140553
   done : 400.000042
   theta0 : 28.588457, theta1 : 0.041746, theta2 : 94.525769
   done : 400.000043
   theta0 : 42.240593, theta1 : 0.096398, theta2 : 51.645989
   done : 400.000043
   theta0 : 98.452431, theta1 : 0.136432, theta2 : 4.831866
   done : 400.000043
   theta0 : 18.022160, theta1 : 0.148059, theta2 : 23.487524
   done : 400.000043
   theta0 : 39.461977, theta1 : 0.097899, theta2 : 51.519412
   done : 400.000042
   theta0 : 40.979868, theta1 : 0.040312, theta2 : 91.401406
   done : 400.000043
   theta0 : 15.466259, theta1 : 0.111276, theta2 : 50.136221
   done : 400.000043
   theta0 : 72.380926, theta1 : 0.013814, theta2 : 99.517853
   done : 400.000043
  </code></pre>