title: Sparse Vector pyspark
body: <p>I'd like to find an efficient method to create spare vectors in PySpark using dataframes.                           </p>
  
  <p>Let's say given the transactional input:</p>
  
  <pre><code>df = spark.createDataFrame([
      (0, "a"),
      (1, "a"),
      (1, "b"),
      (1, "c"),
      (2, "a"),
      (2, "b"),
      (2, "b"),
      (2, "b"),
      (2, "c"),
      (0, "a"),
      (1, "b"),
      (1, "b"),
      (2, "cc"),
      (3, "a"),
      (4, "a"),
      (5, "c")
  ], ["id", "category"])
  </code></pre>
  
  <pre class="lang-none prettyprint-override"><code>+---+--------+
  | id|category|
  +---+--------+
  |  0|       a|
  |  1|       a|
  |  1|       b|
  |  1|       c|
  |  2|       a|
  |  2|       b|
  |  2|       b|
  |  2|       b|
  |  2|       c|
  |  0|       a|
  |  1|       b|
  |  1|       b|
  |  2|      cc|
  |  3|       a|
  |  4|       a|
  |  5|       c|
  +---+--------+
  </code></pre>
  
  <p>In a summed up format:</p>
  
  <pre><code>df.groupBy(df["id"],df["category"]).count().show()
  </code></pre>
  
  <pre class="lang-none prettyprint-override"><code>+---+--------+-----+
  | id|category|count|
  +---+--------+-----+
  |  1|       b|    3|
  |  1|       a|    1|
  |  1|       c|    1|
  |  2|      cc|    1|
  |  2|       c|    1|
  |  2|       a|    1|
  |  1|       a|    1|
  |  0|       a|    2|
  +---+--------+-----+
  </code></pre>
  
  <p>My aim is to get this output by id:</p>
  
  <pre><code>+---+-----------------------------------------------+
  | id|                                       feature |
  +---+-----------------------------------------------+
  |  2|SparseVector({a: 1.0, b: 3.0, c: 1.0, cc: 1.0})|
  </code></pre>
  
  <p>Could you please point me in the right direction? With mapreduce in Java it seemed to be way easier for me. </p>