title: Convert multiple array of structs columns in pyspark sql
body: <p>I have pyspark dataframe with multiple columns (Around 30) of nested structs, that I want to write into csv. (struct
  
  <p>In order to do it, I want to stringify all of the struct columns. </p>
  
  <p>I've checked several answers here:</p>
  
  <p><a href="https://stackoverflow.com/questions/57381557/pyspark-converting-an-array-of-struct-into-string">Pyspark converting an array of struct into string</a></p>
  
  <p><a href="https://stackoverflow.com/questions/47616606/pyspark-dataframe-convert-struct-to-array">PySpark: DataFrame - Convert Struct to Array</a></p>
  
  <p><a href="https://stackoverflow.com/questions/54343635/pyspark-convert-struct-field-inside-array-to-string">PySpark convert struct field inside array to string</a></p>
  
  <p>This is the structure of my dataframe (with around 30 complex keys): </p>
  
  <pre><code>root  
   |-- 1_simple_key: string (nullable = true)  
   |-- 2_simple_key: string (nullable = true)  
   |-- 3_complex_key: struct (nullable = true)  
   |    |-- n1: string (nullable = true)  
   |    |-- n2: struct (nullable = true)  
   |    |    |-- n3: boolean (nullable = true)  
   |    |    |-- n4: boolean (nullable = true)  
   |    |    |-- n5: boolean (nullable = true)  
   |    |-- n6: long (nullable = true)  
   |    |-- n7: long (nullable = true)  
   |-- 4_complex_key: struct (nullable = true)  
   |    |-- n1: string (nullable = true)  
   |    |-- n2: struct (nullable = true)  
   |    |    |-- n3: boolean (nullable = true)  
   |    |    |-- n4: boolean (nullable = true)  
   |    |    |-- n5: boolean (nullable = true)  
   |    |-- n6: long (nullable = true)  
   |    |-- n7: long (nullable = true)  
   |-- 5_complex_key: struct (nullable = true)  
   |    |-- n1: string (nullable = true)  
   |    |-- n2: struct (nullable = true)  
   |    |    |-- n3: boolean (nullable = true)  
   |    |    |-- n4: boolean (nullable = true)  
   |    |    |-- n5: boolean (nullable = true)  
   |    |-- n6: long (nullable = true)  
   |    |-- n7: long (nullable = true)  
  </code></pre>
  
  <p>The proposed solutions are for a single column, and I can't adopt it to multiple columns. </p>
  
  <p>I want to do something of this type:<br>
  1. For each struct_column:<br>
  2. col = stringify(struct_column) </p>
  
  <p>I don't mind creating an additional dataframe for it. I just need to make it ready for csv writing. </p>
  
  <p>Minimal reproducible example:</p>
  
  <pre><code>from pyspark.sql import Row
  d = d = {'1_complex_key': {0: Row(type='1_complex_key', s=Row(n1=False, n2=False, n3=True), x=954, y=238), 1: Row(type='1_complex_key', s=Row(n1=False, n2=False, n3=True), x=956, y=250), 2: Row(type='1_complex_key', s=Row(n1=True, n2=False, n3=False), x=886, y=269)}, '2_complex_key': {0: Row(type='2_complex_key', s=Row(n1=False, n2=False, n3=True), x=901, y=235), 1: Row(type='2_complex_key', s=Row(n1=False, n2=False, n3=True), x=905, y=249), 2: Row(type='2_complex_key', s=Row(n1=False, n2=False, n3=True), x=868, y=270)}, '3_complex_key': {0: Row(type='3_complex_key', s=Row(n1=True, n2=False, n3=False), x=925, y=197), 1: Row(type='3_complex_key', s=Row(n1=False, n2=False, n3=True), x=928, y=206), 2: Row(type='3_complex_key', s=Row(n1=False, n2=False, n3=True), x=883, y=236)}}
  df = pd.DataFrame.from_dict(d)
  spark.conf.set("spark.sql.execution.arrow.enabled", "true")
  s_df = spark.createDataFrame(df)
  s_df.printSchema()
  s_df.write.csv('it_doesnt_write.csv')
  </code></pre>
  
  <p><a href="https://i.stack.imgur.com/AyU6o.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/AyU6o.png" alt="enter image description here"></a>
  <a href="https://i.stack.imgur.com/jA9eo.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/jA9eo.png" alt="enter image description here"></a></p>
  
  <p>So - to summarize: 
  I have a spark dataframe that I want to write to CSV.
  I can't write it to CSV because:</p>
  
  <pre><code>'CSV data source does not support struct&lt;s:struct&lt;n1:boolean,n2:boolean,n3:boolean&gt;,type:string,x:bigint,y:bigint&gt; data type.;'
  </code></pre>
  
  <p>So I want to perform some actions / reversible transformations on this dataframe so that I can write it to CSV, and later read it from the CSV and make it a spark dataframe with the same schema. </p>
  
  <p>How can I do it? Thanks</p>