title: Spark Configuration: SPARK_MEM vs. SPARK_WORKER_MEMORY
body: <p>In <code>spark-env.sh</code>, it's possible to configure the following environment variables:</p>
  <pre class="lang-sh prettyprint-override"><code># - SPARK_WORKER_MEMORY, to set how much memory to use (e.g. 1000m, 2g)
  export SPARK_WORKER_MEMORY=22g
  [...]
  # - SPARK_MEM, to change the amount of memory used per node (this should
  #   be in the same format as the JVM's -Xmx option, e.g. 300m or 1g)
  export SPARK_MEM=3g
  </code></pre>
  <p>If I start a standalone cluster with this:</p>
  <pre class="lang-sh prettyprint-override"><code>$SPARK_HOME/bin/start-all.sh
  </code></pre>
  <p>I can see at the Spark Master UI webpage that all the workers start with only 3GB RAM:</p>
  <pre><code>-- Workers Memory Column --
  22.0 GB (3.0 GB Used)
  22.0 GB (3.0 GB Used)
  22.0 GB (3.0 GB Used)
  [...]
  </code></pre>
  <p>However, I specified <code>22g</code> as <code>SPARK_WORKER_MEMORY</code> in <code>spark-env.sh</code></p>
  <p>I'm somewhat confused by this. Probably I don't understand the difference between &quot;node&quot; and &quot;worker&quot;.</p>
  <p>Can someone explain the difference between the two memory settings and what I might have done wrong?</p>
  <p>I'm using spark-0.7.0. See also <a href="https://spark.apache.org/docs/0.7.0/configuration.html" rel="nofollow noreferrer">here</a> for more configuration info.</p>