title: Spark SparkOutOfMemoryError despite having each task memory greater than input
body: <p>I am using 10 r4.8xlarge with 32 codes and 244 GB memory and 100GB SSD my input is 100 22GB txt files.</p>
  
  <p>Below is my executor parameters</p>
  
  <pre><code> --num-executors 59  --executor-cores 5 --executor-memory 37g
  </code></pre>
  
  <p>I put executor-cores 5 because it seems like people recommend default to 5, and num-executors is <code>floor(32/5) * 10 - 1</code> executor-memory  is <code>(244 / floor(32/5)) (1 - 0.07)</code> as recommended in this tuning <a href="https://blog.cloudera.com/how-to-tune-your-apache-spark-jobs-part-2/" rel="nofollow noreferrer">tutorial</a></p>
  
  <p>Using the above configuration I am getting </p>
  
  <pre><code>org.apache.spark.memory.SparkOutOfMemoryError: error while calling spill() on 
  org.apache.spark.shuffle.sort.ShuffleExternalSorter@5eced6cb : No space left on device
  </code></pre>
  
  <p>Which correspond to this code</p>
  
  <pre><code>    return spark.sparkContext
      .textFile(myInputDir)
      .repartition(15800) // experimentally tested to be a good partition base on 1 file, 1 file the best repartition is 3000
  </code></pre>
  
  <p>Since each task get 37GB and that is more than 22GB, I am a bit confuse on how spark can reach <code>SparkOutOfMemoryError</code> as its just simply reading it and repartitioning it to other nodes.</p>
  
  <p>Can someone point me a direction?</p>