title: How to extract only certain attribute levels from a nested structure in a spark dataframe
body: <p>We want to break a nested data structure into separate entities with Spark &amp; Scala.
  The structure is like:</p>
  
  <pre><code>root
   |-- timestamp: string (nullable = true)
   |-- contract: struct (nullable = true)
   |    |-- category: string (nullable = true)
   |    |-- contractId: array (nullable = true)
   |    |-- items: array (nullable = true)
   |    |    |-- element: struct (containsNull = true)
   |    |    |    |-- active: boolean (nullable = true)
   |    |    |    |-- itemId: string (nullable = true)
   |    |    |    |-- subItems: array (nullable = true)
   |    |    |    |    |-- element: struct (containsNull = true)
   |    |    |    |    |    |-- elementId: string (nullable = true)
  </code></pre>
  
  <p>We want to have contracts, items and subItems in separate data collections. The sub-entities should contain references to their parent, and the top-level fields (timestamp) as audit fields.</p>
  
  <p>Contracts:</p>
  
  <ul>
  <li>auditTimestamp</li>
  <li>category</li>
  <li>contractId</li>
  </ul>
  
  <p>Items:</p>
  
  <ul>
  <li>auditTimestamp</li>
  <li>contractId (foreignKey)</li>
  <li>active</li>
  <li>itemId</li>
  </ul>
  
  <p>SubItems:</p>
  
  <ul>
  <li>auditTimestamp</li>
  <li>itemId (foreignKey)</li>
  <li>elementID</li>
  </ul>
  
  <p>We don't want to configure all the necessary attributes specifically, but only the respective parent attribute to extract, the foreignKey (reference), and what should NOT be extracted (e.g. contract should not contain items, an item should not contain subelements).</p>
  
  <p>We tried with <code>dataframe.select("*").select(explode("contract.*"))</code> and the likes, but we can't make it.
  Any ideas on how to do this elegantly are welcome.</p>
  
  <p>Best
  Alex</p>