title: Alternative to Union in pyspark
body: <p>I have written a snippet to do the following:
  1. Take n rows for each strata from a dataframe (df1)
  2. Rank order the rows by strata
  3. Replace data in one of the columns with data from another data frame (df2)
  4. Union both the dataframe (df1 and df2)</p>
  
  <p>I understand that <code>unionall</code> is an expensive operation in spark. Is there an alternative/more efficient and faster way of doing the same. Thanks </p>
  
  <pre><code>SeedWindow = Window.orderBy("SeedEmail")
  AlphaOutputWindow = Window.partitionBy("col1").orderBy("col2")
  
  seedEmails = (seeds.filter(pos_filter_cond).select("col1....col2")
          .distinct().withColumn("row_id",row_number().over(SeedWindow)))
  
  seedCounts = seedEmails.count()
  
  sampleForSeed = (final_result.withColumn("row_id",row_number().over(AlphaOutputWindow))
          .filter("row_id &lt;= "+str(seedCounts))
      )
  
  sampleAfterSeed = (sampleForSeed.join(seedEmails, ["cols"], "inner"))
  
  finalOutputColumns = [col for col in final_result_moduleCount.columns]
  
  final_result_moduleCount = final_result_moduleCount.select(finalOutputColumns).unionAll(sampleAfterSeed.select(finalOutputColumns))
  </code></pre>