title: Shuffle files cleanup in Spark with externalShuffle service
body: <p>We are using Spark 3.0.1 (standalone mode) with dynamic allocation and external shuffle service.</p>
  <p>After switching to dedicated persistent disks we started getting &quot;out of disk space errors&quot;, so we looked into the /tmp folder and noticed many older application shuffle files still exists there, which is somewhat understandable as shuffle files should stay available as long as the worker is alive.</p>
  <p>But at what point do these files gets deleted? I would assume that on job completion the shuffle files are no longer needed and can (and should) be deleted, but looking at spark code base (ExternalShuffleBlockResolver) it shows the directory cleaner is triggered only after:</p>
  <ul>
  <li>applicationRemoved - deletes entire executor dir (shuffle files included)</li>
  <li>executorRemoved - deletes only non-shuffle files</li>
  </ul>
  <p>Am I missing anything? what service should cleanup shuffle files of completed/failed jobs?</p>
  <p>Note that we use the multi-tenant architecture, meaning our spark app acts as a server to multiple requests.</p>