title: Casting string to ArrayType(DoubleType) pyspark dataframe
body: <p>I have a dataframe in spark with the following schema:
  schema: </p>
  
  <pre><code>StructType(List(StructField(id,StringType,true),
  StructField(daily_id,StringType,true),
  StructField(activity,StringType,true)))
  </code></pre>
  
  <p>Column activity is a String, sample content:</p>
  
  <p><strong>{1.33,0.567,1.897,0,0.78}</strong></p>
  
  <p>I need to cast column Activity to a ArrayType(DoubleType)</p>
  
  <p>In order to get that done i have run the following command:</p>
  
  <pre><code>df = df.withColumn("activity",split(col("activity"),",\s*").cast(ArrayType(DoubleType())))
  </code></pre>
  
  <p>The new schema of the dataframe changed accordingly:</p>
  
  <pre><code>StructType(List(StructField(id,StringType,true),
  StructField(daily_id,StringType,true),
  StructField(activity,ArrayType(DoubleType,true),true)))
  </code></pre>
  
  <p>However, the data now looks like this:
  <strong>[NULL,0.567,1.897,0,NULL]</strong></p>
  
  <p>It changed the first and last element of the array of strings to NULL. I can't figure out why Spark is doing this with the dataframe.</p>
  
  <p>Can please help here on what is the issue?</p>
  
  <p>Many Thanks</p>