title: How to get Top N most similar items based on cosine similarity results in a dataframe?
body: <p>I have a dataset of images ( id, url, features ) on which I performed a Cosine Similarity between all images. The result is a pyspark dataframe with the following structure :</p>
  
  <pre><code>&gt;&gt;&gt; cos_df.printSchema()
  root
   |-- id: integer (nullable = true)
   |-- url: string (nullable = true)
   |-- vec: vector (nullable = true)
  </code></pre>
  
  <p>vec is a column that contains the results of the cosine similarity ( DenseVector ).
  What I'm trying to do is to create a column "similar_urls" OR update "vec" and input, for each row, the top N most similar items based on the vec column values.</p>
  
  <p>For example, if I take id = 26, I want to look in "vec" and find the indexes of the top N items ( id and indexes are the same ) and replace the values of vec with a list of the url of the top N items.</p>
  
  <p>What I tried to do is to :</p>
  
  <ol>
  <li>replace "vec" with a list/array of top N most similar items' indexes (udf)</li>
  <li>replace that list/array with a list/array of urls (udf)</li>
  </ol>
  
  <p>I'm stuck at the first step because I can't seem to transform my "vec" values into an array/list to find the top 10 values.</p>
  
  <pre><code>from pyspark.sql.functions import udf
  
  def convert_to_array(vec):
      return type(vec)
  
  test_udf = udf(convert_to_array, StringType())
  
  cos_df = cos_df.withColumn("vec", test_udf("vec"))
  </code></pre>
  
  <p>When I try to look at the type of vec values, it returns</p>
  
  <pre><code>net.razorvine.pickle.objects.ClassDictConstructor@2db673eb
  </code></pre>
  
  <p>Do you have any idea of what type is that and how can I manipulate it so that I can transform vec ?</p>
  
  <p>P.S : I am also open to any other solution that would be better for the given problem!</p>