title: Splitting an Pyspark RDD into Different columns and convert to Dataframe
body: <p>I have an rdd:</p>
  
  <pre><code>a,1,2,3,4
  b,4,6
  c,8,9,10,11
  </code></pre>
  
  <p>I want to convert this into Spark Data Frame with index:</p>
  
  <pre><code>df:
  
  Index  Name  Number
   0      a     1,2,3,4
   1      b     4,6
   2      c     8,9,10,11
  </code></pre>
  
  <p>I tried splitting the RDD:</p>
  
  <pre><code>parts = rdd.flatMap(lambda x: x.split(","))
  </code></pre>
  
  <p>But that resulted in :</p>
  
  <pre><code>a,
  1,
  2,
  3,...
  </code></pre>
  
  <p>How do I split and convert the RDD to Dataframe in pyspark such that, the first element is taken as first column, and the rest elements combined to a single column ?</p>
  
  <p>As mentioned in the solution:</p>
  
  <pre><code>rd = rd1.map(lambda x: x.split("," , 1) ).zipWithIndex()
  rd.take(3)
  </code></pre>
  
  <p>Output:</p>
  
  <pre><code>[(['a', '1,2,3,4'], 0),
  (['b', '4,6'], 1),
  (['c', '8,9,10,11'], 2)]
  </code></pre>
  
  <p>Next Step:</p>
  
  <pre><code>rd2=rd2=rd.map(lambda x,y: (y, x[0] , x[1]) ).toDF(["index", "name" ,"number"])
  rd2.collect()
  </code></pre>
  
  <p>I am getting the following error:</p>
  
  <pre><code> An error occurred while calling 
  z:org.apache.spark.api.python.PythonRDD.runJob.
  : org.apache.spark.SparkException: Job aborted due to stage failure: 
  Task 0 in stage 55.0 failed 1 times, most recent failure: Lost task 0.0 
  in stage 55.0 (TID 85, localhost, executor driver): 
  org.apache.spark.api.python.PythonException: Traceback (most recent 
  call last):
  </code></pre>
  
  <p>Is it an issue with the version?</p>