title: Why does Spark running in Google Dataproc store temporary files on external storage (GCS) instead of local disk or HDFS while using saveAsTextFile?
body: <p>I have run the following PySpark code:</p>
  
  <pre><code>from pyspark import SparkContext
  
  sc = SparkContext()
  
  data = sc.textFile('gs://bucket-name/input_blob_path')
  sorted_data = data.sortBy(lambda x: sort_criteria(x))
  sorted_data.saveAsTextFile(
      'gs://bucket-name/output_blob_path',
      compressionCodecClass="org.apache.hadoop.io.compress.GzipCodec"
  )
  </code></pre>
  
  <p>Job finished successfully. However, during the job execution Spark created many temporary blobs in the following path <code>gs://bucket-name/output_blob_path/_temporary/0/</code>. I realised that removing of all these temporary blobs at the end took half of the job execution time and CPU utilisation was on 1% during this time (huge waste of resources). </p>
  
  <p>Is there a way to store temporary files on local drive (or HDFS) instead of GCP? I would still like to persist final results (sorted dataset) to GCP.</p>
  
  <p>We were using Dataproc Spark cluster (VM type 16cores, 60GM) with 10 worker nodes. The volume of the input data was 10TB.</p>