title: Using Apache-Spark, reduce or fold an RDD depending on a condition
body: <p>I'm working with Apache Spark and Scala. I have an RDD of String,Int </p>
  
  
  
  <pre class="lang-scala prettyprint-override"><code>val counts =words.map(word =&gt; (word, 1)).reduceByKey((a,b) =&gt; (a + b))     
  </code></pre>
  
  <p>Now I reduced the RDD by Key, but I'd like to add another feature to reduce also the words that are similar.</p>
  
  <p>I though to use Levenshtein distance, Euclidean distance or cosine distance.</p>
  
  <p>So, how can I apply one of this functions to reduce my RDD?</p>
  
  <p>Example: </p>
  
  <pre class="lang-scala prettyprint-override"><code>RDD -&gt;  (forks,12), (fork,4), (chair,15) , (table,1), (tables,11)
  </code></pre>
  
  <p>Admitting that the similarity algorithm works, how can I obtain a reduced RDD like:</p>
  
  <pre class="lang-scala prettyprint-override"><code>RDD -&gt; (fork,16), (table,12), (chair,15)
  </code></pre>
  
  <p>I tried something like:</p>
  
  <pre class="lang-scala prettyprint-override"><code>counts.foldLeft(){(x,y) =&gt; 
    if(x._1.euclideanDistance(y._1) &gt; 0.9) 
      (x,x._2+y._2) 
  }
  </code></pre>