title: Using Python's reduce() to join multiple PySpark DataFrames
body: <p>Does anyone know why using Python3's <code>functools.reduce()</code> would lead to worse performance when joining multiple PySpark DataFrames than just iteratively joining the same DataFrames using a <code>for</code> loop? Specifically, this gives a massive slowdown followed by an out-of-memory error:</p>
  
  <pre><code>def join_dataframes(list_of_join_columns, left_df, right_df):
      return left_df.join(right_df, on=list_of_join_columns)
  
  joined_df = functools.reduce(
      functools.partial(join_dataframes, list_of_join_columns), list_of_dataframes,
  )
  </code></pre>
  
  <p>whereas this one doesn't:</p>
  
  <pre><code>joined_df = list_of_dataframes[0]
  joined_df.cache()
  for right_df in list_of_dataframes[1:]:
      joined_df = joined_df.join(right_df, on=list_of_join_columns)
  </code></pre>
  
  <p>Any ideas would be greatly appreciated. Thanks!</p>