title: Spark: Process multiline input blob
body: <p>I'm new to Hadoop/Spark and trying to process a multiple line input blob into a csv or tab delimited format for further processing. </p>
  
  <p>Example Input </p>
  
  <pre><code>------------------------------------------------------------------------
  AAA=someValueAAA1
  BBB=someValueBBB1
  CCC=someValueCCC1
  DDD=someValueDDD1
  EEE=someValueEEE1
  FFF=someValueFFF1
  ENDOFRECORD
  ------------------------------------------------------------------------
  AAA=someValueAAA2
  BBB=someValueBBB2
  CCC=someValueCCC2
  DDD=someValueDDD2
  EEE=someValueEEE2
  FFF=someValueFFF2
  ENDOFRECORD
  ------------------------------------------------------------------------
  AAA=someValueAAA3
  BBB=someValueBBB3
  CCC=someValueCCC3
  DDD=someValueDDD3
  EEE=someValueEEE3
  FFF=someValueFFF3
  GGG=someValueGGG3
  HHH=someValueHHH3
  ENDOFRECORD
  ------------------------------------------------------------------------
  </code></pre>
  
  <p>Needed output </p>
  
  <pre><code>someValueAAA1, someValueBBB1, someValueCCC1, someValueDDD1, someValueEEE1, someValueFFF1
  someValueAAA2, someValueBBB2, someValueCCC2, someValueDDD2, someValueEEE2, someValueFFF2
  someValueAAA3, someValueBBB3, someValueCCC3, someValueDDD3, someValueEEE3, someValueFFF3
  </code></pre>
  
  <p>Code ive tried so far - </p>
  
  <pre><code>#inputRDD
  val inputRDD = sc.textFile("/somePath/someFile.gz")
  
  #transform
  val singleRDD = inputRDD.map(x=&gt;x.split("ENDOFRECORD")).filter(x=&gt;x.trim.startsWith("AAA"))
  
  
  val logData = singleRDD.map(x=&gt;{
    val rowData = x.split("\n")
  
    var AAA = ""
    var BBB = ""
    var CCC = ""
    var DDD = ""
    var EEE = ""
    var FFF = ""
  
    for (data &lt;- rowData){
      if(data.trim().startsWith("AAA")){
        AAA = data.split("AAA=")(1)
      }else if(data.trim().startsWith("BBB")){
        BBB = data.split("BBB=")(1)
      }else if(data.trim().startsWith("CCC=")){
        CCC = data.split("CCC=")(1)
      }else if(data.trim().startsWith("DDD=")){
        DDD = data.split("DDD=")(1)
      }else if(data.trim().startsWith("EEE=")){
        EEE = data.split("EEE=")(1)
      }else if(data.trim().startsWith("FFF=")){
        FFF = data.split("FFF=")(1)
      }
    }
    (AAA,BBB,CCC,DDD,EEE,FFF)
  })
  
  logData.take(10).foreach(println)
  </code></pre>
  
  <p>This does not seem to work and i get o/p such as </p>
  
  <pre><code>AAA,,,,,,
  ,BBB,,,,,
  ,,CCC,,,,
  ,,,DDD,,,
  </code></pre>
  
  <p>Cant seem to figure out whats wrong here. Do i have to write a custom input format to solve this?</p>