title: Nested dynamic schema not working while parsing JSON using pyspark
body: <p>I am trying to extract certain parameters from a nested JSON <strong>(having dynamic schema)</strong> and generate a spark dataframe using pyspark.</p>
  
  <p>My code works perfectly for level 1 (key:value) but fails get independent columns for each (key:value) pair that are a part of nested JSON.  </p>
  
  <h3>JSON schema sample</h3>
  
  <p>Note - This is not the exact schema. Its just to give the idea of nested nature of the schema</p>
  
  <pre><code>{
    "tweet": {
      "text": "RT @author original message"
      "user": {
            "screen_name": "Retweeter"
      },
      "retweeted_status": {
        "text": "original message".
          "user": {         
              "screen_name": "OriginalTweeter"
          },
          "place": {          
          },
          "entities": {          
          },
          "extended_entities": {          
          }
        },     
      },
      "entities": {      
      },
      "extended_entities": {      
      }
    }
  }
  </code></pre>
  
  <h3>PySpark Code</h3>
  
  <pre><code>from pyspark.sql.types import StructType, StructField, StringType
  
  schema = StructType([
  StructField("text", StringType(), True),
  StructField("created_at", StringType(), True),
  StructField("retweeted_status", StructType([
  StructField("text", StringType(), True),
  StructField("created_at", StringType(), True)]))
  ])
  
  df = spark.read.schema(schema).json("/user/sagarp/NaMo/data/NaMo2019-02-12_00H.json")
  df.show()
  </code></pre>
  
  <h3>Current output - (with real JSON data)</h3>
  
  <p>All (keys:values) under nested <strong>retweet_status</strong> JSON are squashed into 1 single list. eg [text, created_at, entities]</p>
  
  <pre><code>+--------------------+--------------------+--------------------+
  |                text|          created_at|    retweeted_status|
  +--------------------+--------------------+--------------------+
  |RT @Hoosier602: @...|Mon Feb 11 19:04:...|[@CLeroyjnr @Gabr...|
  |RT @EgSophie: Oh ...|Mon Feb 11 19:04:...|[Oh cool so do yo...|
  |RT @JacobAWohl: @...|Mon Feb 11 19:04:...|[@realDonaldTrump...|
  </code></pre>
  
  <h3>Expected output</h3>
  
  <p>I want independent columns for each key. Also, note that you already have a parent level key by the same name <em>text</em>. How will you deal with such instances?</p>
  
  <p>Ideally, I would want columns like "text", "entities", "retweet_status_text", "retweet_status_entities", etc</p>