title: In Apache Spark, why does RDD.union not preserve the partitioner?
body: <p>As everyone knows partitioners in Spark have a huge performance impact on any "wide" operations, so it's usually customized in operations. I was experimenting with the following code:</p>
  
  <pre><code>val rdd1 =
    sc.parallelize(1 to 50).keyBy(_ % 10)
      .partitionBy(new HashPartitioner(10))
  val rdd2 =
    sc.parallelize(200 to 230).keyBy(_ % 13)
  
  val cogrouped = rdd1.cogroup(rdd2)
  println("cogrouped: " + cogrouped.partitioner)
  
  val unioned = rdd1.union(rdd2)
  println("union: " + unioned.partitioner)
  </code></pre>
  
  <p>I see that by default <code>cogroup()</code> always yields an RDD with the customized partitioner, but <code>union()</code> doesn't, it will always revert back to default. This is counterintuitive as we usually assume that a PairRDD should use its first element as partition key. Is there a way to "force" Spark to merge 2 PairRDDs to use the same partition key?</p>