title: Convert a Spark Vector of features into an array
body: <p>I have a features column which is packaged into a Vector of vectors using Spark's VectorAssembler, as follows. <code>data</code> is the input DataFrame (of type <code>spark.sql.DataFrame</code>).</p>
  
  <pre><code>val featureCols = Array("feature_1","feature_2","feature_3")
  val featureAssembler = new VectorAssembler().setInputCols(featureCols).setOutputCol("features")
  val dataWithFeatures = featureAssembler.transform(data)
  </code></pre>
  
  <p>I am developing a custom Classifier using the <code>Classifier</code> and <code>ClassificationModel</code> developer API. <code>ClassificationModel</code> requires development of a <code>predictRaw()</code> function which outputs a vector of predicted labels from the model. </p>
  
  <pre><code>def predictRaw(features: FeaturesType) : Vector
  </code></pre>
  
  <p>This function is set by the API and takes a parameter, features of <code>FeaturesType</code> and outputs a Vector (which in my case I'm taking to be a Spark <code>DenseVector</code> as <code>DenseVector</code> extends the <code>Vector</code> trait). </p>
  
  <p>Due to the packaging by VectorAssembler, the <code>features</code> column is of type <code>Vector</code> and each element is itself a vector, of the original features for each training sample. For example: </p>
  
  <p>features column - of type Vector<br>
  [1.0, 2.0, 3.0] - element1, itself a vector<br>
  [3.5, 4.5, 5.5] - element2, itself a vector</p>
  
  <p>I need to extract these features into an <code>Array[Double]</code> in order to implement my <code>predictRaw()</code> logic. Ideally I would like the following result in order to preserve the cardinality: </p>
  
  <pre><code>`val result: Array[Double] = Array(1.0, 3.5, 2.0, 4.5, 3.0, 4.5)` 
  </code></pre>
  
  <p>i.e. in column-major order as I will turn this into a matrix.</p>
  
  <p>I've tried:</p>
  
  <pre><code>val array = features.toArray // this gives an array of vectors and doesn't work
  </code></pre>
  
  <p>I've also tried to input the features as a DataFrame object rather than a Vector but the API is expecting a Vector, due to the packaging of the features from VectorAssembler. For example, this function inherently works, but doesn't conform to the API as it's expecting FeaturesType to be Vector as opposed to DataFrame:</p>
  
  <pre><code>def predictRaw(features: DataFrame) :DenseVector = {
    val featuresArray: Array[Double] = features.rdd.map(r =&gt; r.getAs[Vector](0).toArray).collect 
  //rest of logic would go here
  }
  </code></pre>
  
  <p>My problem is that <code>features</code> is of type <code>Vector</code>, not <code>DataFrame</code>. The other option might be to package <code>features</code> as a <code>DataFrame</code> but I don't know how to do that without using <code>VectorAssembler</code>.</p>
  
  <p>All suggestions appreciated, thanks! I have looked at <a href="https://stackoverflow.com/questions/44425159/access-element-of-a-vector-in-a-spark-dataframe-logistic-regression-probability">Access element of a vector in a Spark DataFrame (Logistic Regression probability vector)</a> but this is in python and I'm using Scala.</p>