title: how to dynamically parse a json column in a dataframe without knowing its schema in spark scala
body: <p>Given a spark dataframe which has a column which may or may not contain nested json . This nested json is dynamic . The end requirement is to break the json and generate a new dataframe with new columns for each keys present in nested json.</p>
  <p>The json is dynamic so the table generated will be dynamic. Please also consider that the dataframe consists of more than 100 million records.</p>
  <p>eg-</p>
  <p><strong>Input</strong></p>
  <pre><code>------------------------------------------------------------------------
  |id         |key     |type      |value 
  
  |f9f          |BUSI    |off         |false                                                                                                                                                                                                          
  |f96          |NAME    |50          |true                                                                                                                                                                                                           
  |f9z          |BANK    |off         |{&quot;Name&quot;:&quot;United School&quot;,&quot;admNumber&quot;:&quot;197108&quot;,&quot;details&quot;:{&quot;code&quot;:&quot;WEREFFW32&quot;,&quot;studentName&quot;:&quot;Abhishek kumar&quot;,&quot;doc&quot;:&quot;certificate&quot;,&quot;admId&quot;:&quot;3424325328&quot;,&quot;stat&quot;:0,&quot;studentDetails&quot;:false} }|
  </code></pre>
  <p><strong>Output</strong> :-</p>
  <pre><code>--------------------------------------------------------------------------------------------------------------------------
  |id   |key    |type     |value  |Name    | admNumber   |code    | studentName  | doc   |admId    |stat   | studentDetails
  +------------------------------------+-----------------+-------------+----------------------------------------------------                              
  
  |f9f  |BUSI    |off     |false  |NULL    |NULL         |NULL    |NULL          |NULL   |NULL     |NULL   |NULL |                                            
  |f96  |NAME    |50      |true   |NULL    |NULL         |NULL    |NULL          |NULL   |NULL     |NULL   |NULL            |                                                                                                                
  |f9z  |BANK    |off     |NULL   |United School    |197108         |WEREFFW32    |Abhishek kumar          |certificate   |3424325328     |0   |false |    
  </code></pre>