title: How to convert a Spark Dataframe column from vector to a set?
body: <p>I need to process a dataset to identify frequent itemsets. So the input column must be a vector. The original column is a string with the items separated by comma, so i did the following:</p>
  
  <pre><code>functions.split(out_1['skills'], ',')
  </code></pre>
  
  <p>The problem is the, for some rows, I have duplicated values in the <code>skills</code> and this is causing an error when trying to identify the frequent itemsets.</p>
  
  <p>I wanted to convert the vector to a set to remove the duplicated elements. Something like this:</p>
  
  <pre><code>functions.to_set(functions.split(out_1['skills'], ','))
  </code></pre>
  
  <p>But I could not find a function to convert a column from vector to set, i.e., there is no <code>to_set</code> function.</p>
  
  <p>How can I accomplish what I want, i.e., remove the duplicated elements from the vector?</p>