title: Spark Executors hang after Out of Memory
body: <p>I have a spark application running on EMR (16 nodes, 1 master, 15 core, r3.2xlarge instances). For spark executor configuration, we use dynamic Allocation. </p>
  
  <p>While loading the data into the RDD, I see that sometimes when there's a huge amount of data (700 Gb), then Spark runs Out of Memory, but it does not fail the App. Rather the app sits there hung. I'm not sure why this happens but here is my theory :-</p>
  
  <ol>
  <li>We use dataframes which might be caching things. </li>
  <li>The spark flag spark.dynamicAllocation.cachedExecutorIdleTimeout is set to infinity</li>
  </ol>
  
  <p>My theory is that it might be caching things while creating dataframes but the cache is never relinquished and this leads to a Spark hang.</p>
  
  <p>There are two solutions</p>
  
  <ol>
  <li>Increase cluster size (worse case)</li>
  <li>Figure out a way to add a timeout to Spark app.</li>
  <li>Programatically kill the EMR step (could not find an API which does this)</li>
  </ol>
  
  <p>Any leads about how to go about it ? </p>