title: What does theta values of gradient descent mean?
body: <p>I have all the components, I just am not quite sure This is my output:</p>
  
  <pre><code>Theta--&gt;: 0.09604203456288299, 1.1864676227195392
  </code></pre>
  
  <p>How do I interpret that? What does it mean?</p>
  
  <p>I essentially just modified the example from <a href="http://berlinbrowndev.blogspot.sg/2011/12/implementing-example-one-from-machine.html" rel="nofollow"><strong>this description</strong></a>. But I'm not sure if it's really applicable to my problem. I'm trying to perform binary classification on a set of documents. The documents are rendered as bag-of-words style feature vectors of the form:</p>
  
  <p>Example:</p>
  
  <pre><code>Document 1 = ["I", "am", "awesome"]
  Document 2 = ["I", "am", "great", "great"]
  </code></pre>
  
  <p>Dictionary is:</p>
  
  <pre><code>["I", "am", "awesome", "great"]
  </code></pre>
  
  <p>So the documents as a vector would look like:</p>
  
  <pre><code>Document 1 = [1, 1, 1, 0]
  Document 2 = [1, 1, 0, 2]
  </code></pre>
  
  <p>This is my gradient descent code:</p>
  
  <pre><code>public static double [] gradientDescent(final double [] theta_in, final double alpha, final int num_iters, double[][] data ) 
  {
      final double m = data.length;   
      double [] theta = theta_in;
      double theta0 = 0;
      double theta1 = 0;
      for (int i = 0; i &lt; num_iters; i++) 
      {                        
          final double sum0 = gradientDescentSumScalar0(theta, alpha, data );
          final double sum1 = gradientDescentSumScalar1(theta, alpha, data);                                   
          theta0 = theta[0] - ( (alpha / m) * sum0 ); 
          theta1 = theta[1] - ( (alpha / m) * sum1 );                        
          theta = new double [] { theta0, theta1 };
      }
      return theta;
  }
  
  
  //data is the feature vector
  //this theta is weight
  protected static double [] matrixMultipleHthetaByX( final double [] theta, double[][] data ) 
  {
      final double [] vector = new double[ data.length ];
      int i = 0;                 
      for (final double [] d : data) 
      {
          vector[i] = (1.0 * theta[0]) + (d[0] * theta[1]);            
          i++;
      } // End of the for // 
      return vector;
  }
  
  
  protected static double gradientDescentSumScalar0(final double [] theta, final double alpha, double[][] data ) 
  {        
      double sum = 0;
      int i = 0;
      final double [] hthetaByXArr = matrixMultipleHthetaByX(theta, data ); 
      for (final double [] d : data) 
      {
          final double X = 1.0;
          final double y = d[1];
          final double hthetaByX = hthetaByXArr[i];    
          sum = sum + ( (hthetaByX - y) * X );
          i++;
      } // End of the for //
      return sum;
  }
  protected static double gradientDescentSumScalar1(final double [] theta, final double alpha, double[][] data ) 
  {        
      double sum = 0;
      int i = 0;
      final double [] hthetaByXArr = matrixMultipleHthetaByX(theta, data );
      for (final double [] d : data) 
      {
          final double X = d[0];
          final double y = d[1];            
          final double hthetaByX = hthetaByXArr[i];         
          sum = sum + ( (hthetaByX - y) * X );
          i++;
      } // End of the for //
      return sum;
  }
  
  public static double [] batchGradientDescent( double [] weights, double[][] data ) 
  {
      /*
       * From tex:
       * \theta_j := \theta_j - \alpha\frac{1}{m} \sum_{i=1}^m ( h_\theta (x^{(i)})
       */
      final double [] theta_in = weights;
      double [] theta = gradientDescent(theta_in, alpha, iterations, data );
      lastTheta = theta;
      System.out.println("Theta--&gt;: " + theta[0] + ", " + theta[1]);
      return theta;
  }
  </code></pre>
  
  <p>I call it like this:</p>
  
  <pre><code>   final int globoDictSize = globoDict.size(); // number of features
  
     double[] weights = new double[globoDictSize + 1];
     for (int i = 0; i &lt; weights.length; i++) 
     {
         //weights[i] = Math.floor(Math.random() * 10000) / 10000;
         //weights[i] = randomNumber(0,1);
         weights[i] = 0.0;
     }
  
  
     int inputSize = trainingPerceptronInput.size();
     double[] outputs = new double[inputSize];
     final double[][] a = Prcptrn_InitOutpt.initializeOutput(trainingPerceptronInput, globoDictSize, outputs, LABEL);
  
  
  
         for (int p = 0; p &lt; inputSize; p++) 
         {
  
             Gradient_Descent.batchGradientDescent( weights, a );
         }
  </code></pre>
  
  <p>How can I verify that this code is doing what I want? Shouldn't it be outputting a predicted label or something? I've heard I can also apply to it an error function, such as <a href="http://en.wikipedia.org/wiki/Hinge_loss" rel="nofollow"><strong>hinge loss</strong></a>, that would come after the call to batch gradient descent as a seperate component, isn't it? </p>