title: Combining Spark schema without duplicates?
body: <p>To process the data I have, I am extracting the schema before, so that when I read the dataset, I provide the schema instead of going through the expensive step of inferring schema.</p>
  
  <p>In order to construct the schema, I need to merge in several different schema into the final schema, so I have been using <code>union (++)</code> and <code>distinct</code> methods, but I keep getting <code>org.apache.spark.sql.AnalysisException: Duplicate column(s)</code> exception. </p>
  
  <p>For example, say we have two schema in the following structure:</p>
  
  <pre><code>val schema1 = StructType(StructField("A", StructType(
      StructField("i", StringType, true) :: Nil
      ), true) :: Nil)
  
  val schema2 = StructType(StructField("A", StructType(
      StructField("i", StringType, true) :: Nil
      ), true) :: Nil)
  
  val schema3 = StructType(StructField("A", StructType(
      StructField("i", StringType, true) ::
      StructField("ii", StringType, true) :: Nil
      ), true) :: Nil)
  
  val final_schema = (schema1 ++ schema2 ++ schema3).distinct
  
  println(final_schema)
  </code></pre>
  
  <p>which outputs: </p>
  
  <pre><code>StructType(
      StructField(A,StructType(
           StructField(i,StringType,true)),true), 
      StructField(A,StructType(
          StructField(i,StringType,true),    
          StructField(ii,StringType,true)),true))
  </code></pre>
  
  <p>I understand that only schema structure that exactly match another schema will get filtered out by <code>distinct</code>. However I want the result to look like this:</p>
  
  <pre><code>StructType(
      StructField(A,StructType(
          StructField(i,StringType,true),    
          StructField(ii,StringType,true)),true))
  </code></pre>
  
  <p>in which all the gets "combined" into one schema. I have sifted through all the methods in <a href="https://spark.apache.org/docs/1.6.0/api/scala/index.html#org.apache.spark.sql.types.StructType" rel="nofollow noreferrer">scala documentation</a> but I cannot seem to find the right method to solve this. Any ideas?</p>
  
  <p>EDIT: </p>
  
  <p>The end goal will be to feed in <code>final_schema</code> to <code>sqlContext.read.schema</code> and read RDD of JSON strings using <code>read</code> method.</p>