title: AWS Glue - Running out of disk space on worker threads while processing data
body: <p>I'm running a glue job on a s3 dataset of around 6 million files totaling to about 80 GB, on which I am performing a window function and writing to a different s3 location. My glue job is using 50 G2.X workers and default Spark partitioning. When I run this, I receive an error listed below. Any suggestions on how to keep from running out of storage on an executor?</p>
  <blockquote>
  <p>scheduler.TaskSetManager (Logging.scala:logWarning(66)): Lost task 401.0 in stage 4.0 (TID 505643, 172.35.178.124, executor 13): org.apache.spark.memory.SparkOutOfMemoryError: error while calling spill() on org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorter@16007c9 : No space left on device
  at org.apache.spark.memory.TaskMemoryManager.acquireExecutionMemory(TaskMemoryManager.java:219)
  at org.apache.spark.memory.TaskMemoryManager.allocatePage(TaskMemoryManager.java:285)
  at org.apache.spark.memory.MemoryConsumer.allocatePage(MemoryConsumer.java:117)
  at org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorter.acquireNewPageIfNecessary(UnsafeExternalSorter.java:383)
  at org.apache.spark.util.collection.unsafe.sort.UnsafeExternalSorter.insertRecord(UnsafeExternalSorter.java:407)
  at org.apache.spark.sql.execution.UnsafeExternalRowSorter.insertRow(UnsafeExternalRowSorter.java:135)
  at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.sort_addToSorter_0$(Unknown Source)
  at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
  at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
  at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$13$$anon$1.hasNext(WholeStageCodegenExec.scala:636)
  at org.apache.spark.sql.execution.window.WindowExec$$anonfun$11$$anon$1.fetchNextRow(WindowExec.scala:314)
  at org.apache.spark.sql.execution.window.WindowExec$$anonfun$11$$anon$1.(WindowExec.scala:323)
  at org.apache.spark.sql.execution.window.WindowExec$$anonfun$11.apply(WindowExec.scala:303)
  at org.apache.spark.sql.execution.window.WindowExec$$anonfun$11.apply(WindowExec.scala:302)
  at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$23.apply(RDD.scala:801)
  at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$23.apply(RDD.scala:801)
  at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
  at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
  at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)</p>
  </blockquote>