title: pyspark: Explode struct into columns
body: <p>I have created an udf that returns a StructType which is not nested. Simply a and array of mixed types (int, float) with field names. I want to explode /split them into separate columns. Note that this will create roughly 50 new columns. Through googling I found this solution:</p>
  
  <pre><code>df_split = df.select('ID', 'my_struct.*')
  </code></pre>
  
  <p>This works. However performance is absolutely terrible, eg. unusable. Checking on the cluster nodes this also uses only 1 core. But this only explains a small part of the issue.</p>
  
  <p>So what would be a good way to achieve my goal and why is above solution so slow?</p>
  
  <p>EDIT:</p>
  
  <p>it seems to be the specific combination of the udf and the splitting that results in the poor performance. This is slow:</p>
  
  <pre><code>df_udf = df.withColumn('udf', my_udf(df.input))
  df_exploded = df_udf.select('input', 'udf.*')
  df_exploded.show(5)
  </code></pre>
  
  <p>This is fast:</p>
  
  <pre><code>df_udf = df.withColumn('udf', my_udf(df.input))
  df_udf.cache()
  df_exploded = df_udf.select('input', 'udf.*')
  df_exploded.show(5)
  </code></pre>