title: How to use to_json and from_json to eliminate nested structfields in pyspark dataframe?
body: <p><a href="https://stackoverflow.com/a/48906217">This solution</a> in theory, works perfectly for what I need, which is to create a new copied version of a dataframe while excluding certain nested structfields. here is a minimally reproducible artifact of my issue:</p>
  
  <pre><code>&gt;&gt;&gt; df.printSchema()
  root
  | -- big: array(nullable=true)
  | | -- element: struct(containsNull=true)
  | | | -- keep: string(nullable=true)
  | | | -- delete: string(nullable=true)
  </code></pre>
  
  <p>which you can instantiate like such:</p>
  
  <pre><code>schema = StructType([StructField("big", ArrayType(StructType([
      StructField("keep", StringType()),
      StructField("delete", StringType())
  ])))])
  df = spark.createDataFrame(spark.sparkContext.emptyRDD(), schema)
  </code></pre>
  
  <p><strong>My goal is to convert the dataframe (along with the values in the columns I want to keep) to one that excludes certain nested structs, like <code>delete</code> for example.</strong></p>
  
  <pre><code>root
  | -- big: array(nullable=true)
  | | -- element: struct(containsNull=true)
  | | | -- keep: string(nullable=true)
  </code></pre>
  
  <p>According to the solution I linked that tries to leverage pyspark.sql's <code>to_json</code> and <code>from_json</code> functions, it should be accomplishable with something like this:</p>
  
  <pre><code>new_schema = StructType([StructField("big", ArrayType(StructType([
               StructField("keep", StringType())
  ])))])
  
  test_df = df.withColumn("big", to_json(col("big"))).withColumn("big", from_json(col("big"), new_schema))
  
  &gt;&gt;&gt; test_df.printSchema()
  root
  | -- big: struct(nullable=true)
  | | -- big: array(nullable=true)
  | | | -- element: struct(containsNull=true)
  | | | | -- keep: string(nullable=true)
  
  &gt;&gt;&gt; test_df.show()
  +----+
  | big|
  +----+
  |null|
  +----+
  </code></pre>
  
  <p>So either I'm not following his directions right, or it doesn't work. How do you do this without a udf?</p>
  
  <p><a href="https://spark.apache.org/docs/2.1.0/api/python/pyspark.sql.html#pyspark.sql.functions.to_json" rel="nofollow noreferrer">Pyspark to_json documentation</a>
  <a href="https://spark.apache.org/docs/2.1.0/api/python/pyspark.sql.html#pyspark.sql.functions.from_json" rel="nofollow noreferrer">Pyspark from_json documentation</a></p>