title: How to read a nested collection in Spark
body: <p>I have a parquet table with one of the columns being </p>
  
  <blockquote>
    <p>, array&lt;struct&lt;col1,col2,..colN>></p>
  </blockquote>
  
  <p>Can run queries against this table in Hive using LATERAL VIEW syntax.</p>
  
  <p>How to read this table into an RDD, and more importantly how to filter, map etc this nested collection in Spark?</p>
  
  <p>Could not find any references to this in Spark documentation. Thanks in advance for any information!</p>
  
  <p>ps. I felt might be helpful to give some stats on the table. 
   Number of columns in main table ~600. Number of rows ~200m.
   Number of "columns" in nested collection ~10. Avg number of records in nested collection ~35. </p>