title: Column operation on Spark RDDs
body: <p>I have a RDD with MANY columns (e.g. hundreds), and most of my operation is on columns, e.g. I need to create many intermediate variables from different columns.</p>
  
  <p>What is the most efficient way to do this?</p>
  
  <p>For example, if my <code>dataRDD[Array[String]]</code> is like below: </p>
  
  <pre><code>123, 523, 534, ..., 893 
  536, 98, 1623, ..., 98472 
  537, 89, 83640, ..., 9265 
  7297, 98364, 9, ..., 735 
  ...... 
  29, 94, 956, ..., 758 
  </code></pre>
  
  <p>I will need to create a new column or a variable as <code>newCol1 = 2ndCol+19thCol</code>, and another new column based on <code>newCol1</code> and the existing columns: <code>newCol2 = function(newCol1, 34thCol)</code>.</p>
  
  <p>What is the best way of doing this?</p>
  
  <p>I have been thinking using index for the intermediate variables and the <code>dataRDD</code>, and then join them together on the index to do my calculation:</p>
  
  <pre><code>var dataRDD = sc.textFile("/test.csv").map(_.split(","))
  val dt = dataRDD.zipWithIndex.map(_.swap)
  val newCol1 = dataRDD.map(x =&gt; x(1)+x(18)).zipWithIndex.map(_.swap)
  val newCol2 = newCol1.join(dt).map(x=&gt; function(.........))
  </code></pre>
  
  <p>Is there a better way of doing this?</p>