title: java.lang.OutOfMemoryError: GC overhead limit exceeded when creating data structure for 1 million elements
body: <p>When I run the code shown below, I get a java.lang.OutOfMemoryError: GC overhead limit exceeded on <code>line 16: svm_node node = new svm_node();</code>. The code is run on an array of ~1 million elements, where each element holds 100 shorts.</p>
  
  <pre><code>// read in a problem (in svmlight format)
  private void read(SupportVector[] vectors) throws IOException
  {
      int length = vectors.length; // Length of training data
      double[] classification = new double[length]; // This is redundant for our one-class SVM.
      svm_node[][] trainingSet = new svm_node[length][]; // The training set.
      for(int i = 0; i &lt; length; i++)
      {
          classification[i] = 1; // Since classifications are redundant in our setup, they all belong to the same class, 1.
  
          // each vector. The vector has to be one index longer than the actual vector,
          // because the implementation needs an empty node in the end with index -1.
          svm_node[] vector = new svm_node[vectors[i].getLength() + 1];
  
          double[] doubles = vectors[i].toDouble(); // The SVM runs on doubles.
          for(int j = 0; j &lt; doubles.length; j++) {
              svm_node node = new svm_node();
              node.index = j;
              node.value = doubles[j];
              vector[j] = node;
          }
          svm_node last = new svm_node();
          last.index = -1;
          vector[vector.length - 1] = last;
  
          trainingSet[i] = vector;
      }
  
      svm_problem problem = new svm_problem();
      problem.l = length;
      problem.y = classification;
      problem.x = trainingSet;
  }
  </code></pre>
  
  <p>From the exception, I guess the garbage collector cannot properly sweep up my new svm_nodes, but I am unable to see how I can optimize my object creation, to avoid creating too many new svn_nodes, that sits helpless in the heap.</p>
  
  <p>I cannot change the data structure, as it is what <a href="http://www.csie.ntu.edu.tw/~r94100/libsvm-2.8/README" rel="nofollow">LIBSVM</a> uses as input to its support vector machine.</p>
  
  <p>My question is: Is this error related to the garbage collector not being able to collect my svm_nodes, or am I simply trying to parse a data structure with too many elements?</p>
  
  <p>PS: I already set the heap size to the maximum for my 32bit application (2gb).</p>