title: dot product of a combination of elements of an RDD using pySpark
body: <p>I have an RDD where each element is a tuple of the form </p>
  
  <pre><code>[ (index1,SparseVector({idx1:1,idx2:1,idx3:1,...})) , (index2,SparseVector() ),... ]
  </code></pre>
  
  <p>I would like to take a dot-product of each of the values in this RDD by using the SparseVector1.dot(SparseVector2) method provided by <code>mllib.linalg.SparseVector</code> class. I am aware that python has an <code>itertools.combinations</code> module that can be used to achieve the combinations of dot-products to be calculated. Could someone provide a code-snippet to achieve the same? I can only thing of doing an <code>RDD.collect()</code> so I receive a list of all elements in the RDD and then running the itertools.combinations on this list but this as per my understanding would perform all the calculations on the root and wouldn't be distributed per-se. Could someone please suggest a more distributed way of achieving this?</p>