title: How to split parquet files into many partitions in Spark?
body: <p>So I have just 1 parquet file I'm reading with Spark (using the SQL stuff) and I'd like it to be processed with 100 partitions.  I've tried setting <code>spark.default.parallelism</code> to 100, we have also tried changing the compression of the parquet to none (from gzip).  No matter what we do the first stage of the spark job only has a single partition (once a shuffle occurs it gets repartitioned into 100 and thereafter obviously things are much much faster).</p>
  
  <p>Now according to a few sources (like below) parquet should be splittable (even if using gzip!), so I'm super confused and would love some advice.</p>
  
  <p><a href="https://www.safaribooksonline.com/library/view/hadoop-application-architectures/9781491910313/ch01.html" rel="noreferrer">https://www.safaribooksonline.com/library/view/hadoop-application-architectures/9781491910313/ch01.html</a></p>
  
  <p>I'm using spark 1.0.0, and apparently the default value for <code>spark.sql.shuffle.partitions</code> is 200, so it can't be that.  In fact all the defaults for parallelism are much more than 1, so I don't understand what's going on.</p>