title: Spark java.lang.OutOfMemoryError: Java heap space
body: <p>My cluster: 1 master, 11 slaves, each node has 6 GB memory.</p>
  
  <p>My settings:</p>
  
  <pre><code>spark.executor.memory=4g, Dspark.akka.frameSize=512
  </code></pre>
  
  <p><strong>Here is the problem:</strong></p>
  
  <p><strong>First</strong>, I read some data (2.19 GB) from HDFS to RDD:</p>
  
  <pre class="lang-scala prettyprint-override"><code>val imageBundleRDD = sc.newAPIHadoopFile(...)
  </code></pre>
  
  <p><strong>Second</strong>, do something on this RDD:</p>
  
  <pre class="lang-scala prettyprint-override"><code>val res = imageBundleRDD.map(data =&gt; {
                                 val desPoints = threeDReconstruction(data._2, bg)
                                   (data._1, desPoints)
                               })
  </code></pre>
  
  <p><strong>Last</strong>, output to HDFS:</p>
  
  <pre class="lang-scala prettyprint-override"><code>res.saveAsNewAPIHadoopFile(...)
  </code></pre>
  
  <p>When I run my program it shows:</p>
  
  <pre><code>.....
  14/01/15 21:42:27 INFO cluster.ClusterTaskSetManager: Starting task 1.0:24 as TID 33 on executor 9: Salve7.Hadoop (NODE_LOCAL)
  14/01/15 21:42:27 INFO cluster.ClusterTaskSetManager: Serialized task 1.0:24 as 30618515 bytes in 210 ms
  14/01/15 21:42:27 INFO cluster.ClusterTaskSetManager: Starting task 1.0:36 as TID 34 on executor 2: Salve11.Hadoop (NODE_LOCAL)
  14/01/15 21:42:28 INFO cluster.ClusterTaskSetManager: Serialized task 1.0:36 as 30618515 bytes in 449 ms
  14/01/15 21:42:28 INFO cluster.ClusterTaskSetManager: Starting task 1.0:32 as TID 35 on executor 7: Salve4.Hadoop (NODE_LOCAL)
  Uncaught error from thread [spark-akka.actor.default-dispatcher-3] shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for ActorSystem[spark]
  java.lang.OutOfMemoryError: Java heap space
  </code></pre>
  
  <p>There are too many tasks?</p>
  
  <p><strong>PS</strong>: Every thing is ok when the input data is about 225 MB. </p>
  
  <p>How can I solve this problem?</p>