title: Making Glue delete source data after a job
body: <p>AWS Glue is great for transforming data from a raw form into whichever format you need, and keeping the source and destination data sets synchronized.</p>
  
  <p>However, I have a scenario where data lands in a 'landing area' bucket from untrusted external sources, and the first ETL step needs to be a data validation step which only allows valid data to pass to the data lake, while non-valid data is moved to a quarantine bucket for manual inspection.</p>
  
  <p>Non-valid data includes:</p>
  
  <ul>
  <li>bad file formats/encodings</li>
  <li>unparseable contents</li>
  <li>mismatched schemas</li>
  <li>even some sanity checks on the data itself</li>
  </ul>
  
  <p>The 'landing area' bucket is not part of the data lake, it is only a temporary dead drop for incoming data, and so I need the validation job to delete the files from this bucket once it has moved them to the lake and/or quarantine buckets.</p>
  
  <p>Is this possible with Glue? If the data is deleted from the source bucket, won't Glue end up removing it downstream in a subsequent update?</p>
  
  <p>Am I going to need a different tool (e.g. StreamSets, NiFi, or Step Functions with AWS Batch) for this validation step, and only use Glue once the data is in the lake?</p>
  
  <p>(I know I can set lifecycle rules on the bucket itself to delete the data after a certain time, like 24 hours, but in theory this could delete data before Glue has processed it, e.g. in case of a problem with the Glue job)</p>