title: Pyspark: Extract Multiclass Classification results as different columns
body: <p>I'm using the RandomForestClassifier object for a multiclass classification problem.
  The output dataframe of the prediction presents the 'probability' columns as a vector:</p>
  
  <pre><code>df.select('probability').printSchema()
  root
   |-- probability: vector (nullable = true)
  </code></pre>
  
  <p>Each row is a vector of 4:</p>
  
  <pre><code>df.select('probability').show(3)
  +--------------------+
  |         probability|
  +--------------------+
  |[0.02753394443688...|
  |[7.95347766409877...|
  |[0.02264704615632...|
  +--------------------+
  </code></pre>
  
  <p>I would like to create 4 columns on my df with one Double value each.</p>
  
  <p>A similar question suggests this solution:</p>
  
  <pre><code>from pyspark.sql.functions import udf
  from pyspark.sql.types import FloatType
  
  firstelement=udf(lambda v:float(v[0]),FloatType())
  df.select(firstelement('probability'))
  </code></pre>
  
  <p>The solution works but when I try to assign the value to a new column with</p>
  
  <pre><code>df.withColumn('prob_SELF', df.select(firstelement('probability'))['&lt;lambda&gt;(probability)'])
  </code></pre>
  
  <p>I have the following error:</p>
  
  <pre><code>AnalysisException: 'resolved attribute(s) &lt;lambda&gt;(probability)#26116 missing from prediction#25521
  </code></pre>