title: Does Spark support Partition Pruning with Parquet Files
body: <p>I am working with a large dataset, that is partitioned by two columns - <code>plant_name</code> and <code>tag_id</code>. The second partition - <code>tag_id</code> has 200000 unique values, and I mostly access the data by specific <code>tag_id</code> values. If I use the following Spark commands:</p>
  
  <pre><code>sqlContext.setConf("spark.sql.hive.metastorePartitionPruning", "true")
  sqlContext.setConf("spark.sql.parquet.filterPushdown", "true")
  val df = sqlContext.sql("select * from tag_data where plant_name='PLANT01' and tag_id='1000'")
  </code></pre>
  
  <p>I would expect a fast response as this resolves to a single partition. In Hive and Presto this takes seconds, however in Spark it runs for hours.</p>
  
  <p>The actual data is held in a S3 bucket, and when I submit the sql query, Spark goes off and first gets all the partitions from the Hive metastore (200000 of them), and then calls <code>refresh()</code> to force a full status list of all these files in the S3 object store (actually calling <code>listLeafFilesInParallel</code>).</p>
  
  <p>It is these two operations that are so expensive, are there any settings that can get Spark to prune the partitions earlier - either during the call to the metadata store, or immediately afterwards? </p>