title: Spark SQL JOIN gives different execution times when using df.repartition versus reading in and using a partitioned Parquet
body: <p>I am doing a Broadcast Hash join of two DataFrames on a common id.</p>
  <p>I noticed that when I try <code>DataFrame.repartition(&quot;id&quot;)</code> and pass the DataFrames into the <code>JOIN</code> query, the execution time is a lot slower than if I write the DataFrames to Parquet files with <code>DataFrame.write.partitionBy(&quot;id&quot;)</code>, then read in the Parquet as a DataFrame before passing it into the query.</p>
  <p>Why is there a difference between repartitioning the DataFrame versus reading in a DataFrame from a partitioned Parquet?</p>
  <p>Both the first and second case have an equal number of partitions when I call <code>df.rdd.getNumPartitions()</code>, and both DataFrames are sorted and partitioned according to the <code>id</code> column.</p>
  <p>For <code>DataFrame.repartition(&quot;id&quot;)</code> the query time is three minutes, taking three jobs with 7 stages altogether.</p>
  <p>For partitioning into Parquet, the query time is one minute, with one job and 1 stage.</p>
  <p>1 dataset is 41 mil rows while the other is 2 mil rows.</p>