title: Why is spark-submit job leaving a process running on cluster (EMR) master node?
body: <p>I am submitting a spark job to Livy through an AWS lambda function. The job runs to the end of the driver program but then does not shutdown.</p>
  
  <p>If spark.stop() or sc.stop() is added to the end of the driver program, the spark job finishes on the YARN resource manager and Livy will report a success. However, there is still a livy process running on the master node which takes around 1.5Gb of memory. If many jobs are submitted this eventually uses and holds all of the master node memory.</p>
  
  <p>The job:</p>
  
  <ul>
  <li><p>Pulls records from a hive table</p></li>
  <li><p>Collects these records on the master node and then writes them to a pdf file using apache pdfbox</p></li>
  <li><p>Uploads the resulting PDF to S3</p></li>
  </ul>
  
  <p>Directly running spark-submit on the cluster produces the same results, however if I ctrl+c whilst the spark-submit job is running, the process on the master node is ended.</p>
  
  <p>We are expecting the job to finish by itself when reaching the end of the driver program. If not this, the shutdown hook should be called when spark.stop() is called.</p>