title: Why would I want .union over .unionAll in Spark for SchemaRDDs?
body: <p>I'm trying to wrap my head around these two functions in the <a href="http://spark.apache.org/docs/1.2.0/api/scala/index.html#org.apache.spark.sql.SchemaRDD" rel="noreferrer">Spark SQL documentation</a>–</p>
  
  <ul>
  <li><p><em>def union(other: RDD[Row]): RDD[Row]</em></p>
  
  <p>Return the union of this RDD and another one.</p></li>
  <li><p><em>def unionAll(otherPlan: SchemaRDD): SchemaRDD</em></p>
  
  <p>Combines the tuples of two RDDs with the same schema, keeping duplicates.</p></li>
  </ul>
  
  <p>This is not the standard behavior of UNION vs UNION ALL, <a href="https://stackoverflow.com/questions/49925/what-is-the-difference-between-union-and-union-all">as documented in this SO question</a>.</p>
  
  <p>My code here, borrowing from the <a href="http://spark.apache.org/docs/1.1.0/sql-programming-guide.html#tab_scala_3" rel="noreferrer">Spark SQL documentation</a>, has the two functions returning the same results.</p>
  
  <pre><code>scala&gt; case class Person(name: String, age: Int)
  scala&gt; import org.apache.spark.sql._
  scala&gt; val one = sc.parallelize(Array(Person("Alpha",1), Person("Beta",2)))
  scala&gt; val two = sc.parallelize(Array(Person("Alpha",1), Person("Beta",2),  Person("Gamma", 3)))
  scala&gt; val schemaString = "name age"
  scala&gt; val schema = StructType(schemaString.split(" ").map(fieldName =&gt; StructField(fieldName, StringType, true)))
  scala&gt; val peopleSchemaRDD1 = sqlContext.applySchema(one, schema)
  scala&gt; val peopleSchemaRDD2 = sqlContext.applySchema(two, schema)
  scala&gt; peopleSchemaRDD1.union(peopleSchemaRDD2).collect
  res34: Array[org.apache.spark.sql.Row] = Array([Alpha,1], [Beta,2], [Alpha,1], [Beta,2], [Gamma,3])
  scala&gt; peopleSchemaRDD1.unionAll(peopleSchemaRDD2).collect
  res35: Array[org.apache.spark.sql.Row] = Array([Alpha,1], [Beta,2], [Alpha,1], [Beta,2], [Gamma,3])
  </code></pre>
  
  <p>Why would I prefer one over the other?</p>