title: Case sensitive parquet schema merge in Spark
body: <p>I'm trying to load and analyze some parquet files with Spark.
  I'm using <code>schemaMerge</code> to load the files since newer files have some extra columns. Also some files have their column names in lower case and others in upper case.</p>
  
  <p>For example</p>
  
  <p><code>file1.parquet</code> has a schema like</p>
  
  <pre><code>column1 integer,
  column2 integer
  </code></pre>
  
  <p>and <code>file2.parquet</code> has something like:</p>
  
  <pre><code>Column1 integer,
  Column2 integer,
  Column3 integer
  </code></pre>
  
  <p>I'm running into an issue with <code>inferSchema</code> method of <code>ParquetFileFormat</code> class. Schema merging is delegated to <code>StructType</code> <code>merge</code> method of spark sql. From what I can tell, that method can only work in a case sensitive way.
  Internally it uses a map to lookup fields by name and if the cases don't match it will interpret that as a new field. Later, when schema is checked for duplicates, case sensitivity configuration is respected and we end up with having duplicate columns. This results in </p>
  
  <pre><code>org.apache.spark.sql.AnalysisException: Found duplicate column(s) in the data schema
  </code></pre>
  
  <p>Is there any way to make schema merge case insensitive?</p>
  
  <p>I was expecting to get something like this as a resulting schema:</p>
  
  <pre><code>column1 integer,
  column2 integer,
  Column3 integer
  </code></pre>