title: Flattening a very nested Spark Scala dataframe
body: <p>I have a very nested dataframe that I'm trying to flatten. The original schema looks like:</p>
  
  <pre><code> |-- _History: struct (nullable = true)
   |    |-- Article: array (nullable = true)
   |    |    |-- element: struct (containsNull = true)
   |    |    |    |-- Id: string (nullable = true)
   |    |    |    |-- Timestamp: long (nullable = true)
   |    |-- Channel: struct (nullable = true)
   |    |    |-- Music: array (nullable = true)
   |    |    |    |-- element: long (containsNull = true)
   |    |    |-- Sports: array (nullable = true)
   |    |    |    |-- element: long (containsNull = true)
   |    |    |-- Style: array (nullable = true)
   |    |    |    |-- element: long (containsNull = true)
  </code></pre>
  
  <p>I'm able to flatten most fields using the recursive function:</p>
  
  <pre><code>implicit class DataFrameFlattener(df: DataFrame) {
    def flattenSchema: DataFrame = {
      df.select(flatten(Nil, df.schema): _*)
    }
  
    protected def flatten(path: Seq[String], schema: DataType): Seq[Column] = schema match {
      case s: StructType =&gt; s.fields.flatMap(f =&gt; flatten(path :+ f.name, f.dataType))
      case other =&gt; col(path.map(n =&gt; s"`$n`").mkString(".")).as(path.mkString(".")) :: Nil
    } 
  }
  </code></pre>
  
  <p>However, this doesn't seem to be able to flatten <code>_History.Article.Id</code> and <code>_History.Article.Timstamp</code> in the schema above. Why is this and how do I flatten these two fields into their own columns within the dataframe? </p>