title: Training Sparks word2vec with a RDD[String]
body: <p>I'm new to Spark and Scala so I might have misunderstood some basic things here. I'm trying to train Sparks word2vec model on my own data. According to their <a href="http://spark.apache.org/docs/latest/mllib-feature-extraction.html" rel="nofollow">documentation</a>, one way to do this is</p>
  
  <pre><code>val input = sc.textFile("text8").map(line =&gt; line.split(" ").toSeq)
  val word2vec = new Word2Vec()
  val model = word2vec.fit(input)
  </code></pre>
  
  <p>The <code>text8</code> dataset contains one line of many words, meaning that <code>input</code> will become an <code>RDD[Seq[String]]</code>.</p>
  
  <p>After massaging my own dataset, which has one word per line, using different <code>map</code>s etc. I'm left with an <code>RDD[String]</code>, but I can't seem to be able to train the word2vec model on it. I tried doing <code>input.map(v =&gt; Seq(v))</code> which does actually give an <code>RDD[Seq[String]]</code>, but that will give one sequence for each word, which I guess is totally wrong.</p>
  
  <p>How can I wrap a sequence around my strings, or is there something else I have missed?</p>
  
  <p><strong>EDIT</strong></p>
  
  <p>So I kind of figured it out. From my <code>clean</code> being an <code>RDD[String]</code> I do <code>val input = sc.parallelize(Seq(clean.collect().toSeq))</code>. This gives me the correct data structure (<code>RDD[Seq[String]]</code>) to fit the word2vec model. However, running collect on a large dataset gives me out of memory error. I'm not quite sure how they intend the fitting to be done? Maybe it is not really parallelizable. Or maybe I'm supposed to have several semi-long sequences of strings inside and RDD, instead of one long sequence like I have now?</p>