title: Python: How is the code for Stochastic Gradient Descent working?
body: <pre><code>n_epochs = 50
  t0, t1 = 5, 50 # learning schedule hyperparameters
  def learning_schedule(t):
  return t0 / (t + t1)
  theta = np.random.randn(2,1) # random initialization
  
  for epoch in range(n_epochs):
     for i in range(m):
        random_index = np.random.randint(m)
        xi = X_b[random_index:random_index+1]
        yi = y[random_index:random_index+1]
        gradients = 2 * xi.T.dot(xi.dot(theta) - yi)
        eta = learning_schedule(epoch * m + i)
        theta = theta - eta * gradients
  </code></pre>
  <blockquote>
  <p>By convention we iterate by rounds of m iterations; each round is called an epoch. While the Batch Gradient Descent code iterated 1,000 times through the whole training set, this code goes through the training set only 50 times and reaches a pretty good solution</p>
  </blockquote>
  <p>The author says that the code iterates through the training set only 50 times. How is that possible?</p>
  <p>Here isn't it the case that for every epoch, 1-&gt;50, i is going from 1-&gt;100 and the training data is being iterated 50*100=5000 times?</p>