title: flattening array of struct in pyspark
body: <p>I have an XML file converted to dataframe using spark-xml package. The dataframe has the following structure:</p>
  
  <pre><code>root
   |-- results: struct (nullable = true)
   |    |-- result: struct (nullable = true)
   |    |    |-- categories: struct (nullable = true)
   |    |    |    |-- category: array (nullable = true)
   |    |    |    |    |-- element: struct (containsNull = true)
   |    |    |    |    |    |-- value: string (nullable = true)
  </code></pre>
  
  <p>if I select the category column(which may appear multiple times under categories):</p>
  
  <pre><code>df.select((col('results.result.categories.category')).alias("result_categories"))
  </code></pre>
  
  <p>For one record, The result will look like</p>
  
  <pre><code>[[result1], [result2]]
  </code></pre>
  
  <p>I'm trying to flatten the results:</p>
  
  <pre><code>[result1, result2]
  </code></pre>
  
  <p>When I use the flatten function, I'm getting an error message:</p>
  
  <pre><code>df.select(flatten(col('results.result.categories.category')).alias("Hits_Category"))
   cannot resolve 'flatten(`results`.`result`.`categories`.`category`)' due to data type mismatch: The argument should be an array of arrays, but '`results`.`result`.`categories`.`category`' is of array&lt;struct&lt;value:string&gt;
  </code></pre>
  
  <p>I end up creating a udf, and pass the column to the udf which spits out a flattened string version of the column. </p>
  
  <p>Is there a better way? </p>