title: How to use AWS Glue / Spark to convert CSVs partitioned and split in S3 to partitioned and split Parquet
body: <p>In AWS Glue's catalog, I have an external table defined with partitions that looks roughly like this in S3 and partitions for new dates are added daily:</p>
  
  <pre><code>s3://my-data-lake/test-table/
      2017/01/01/
          part-0000-blah.csv.gz
          .
          .
          part-8000-blah.csv.gz
      2017/01/02/
          part-0000-blah.csv.gz
          .
          .
          part-7666-blah.csv.gz
  </code></pre>
  
  <p>How could I use Glue/Spark to convert this to parquet that is also partitioned by date and split across <em>n</em> files per day?. The examples don't cover partitioning or splitting or provisioning (how many nodes and how big). Each day contains a couple hundred GBs.</p>
  
  <p>Because the source CSVs are not necessarily in the right partitions (wrong date) and are inconsistent in size, I'm hoping to to write to partitioned parquet with the right partition and more consistent size. </p>