title: Functional approach for a transaction processing
body: <p>I need to create a transactional file by comparing 2 files (old vs new). If the key is in old file and not in new file, I have to populate an indicator ‘D’ at the end of the record and if the key is in new file and not in old file, the record should get indicator ‘A’ and if the keys are matched and there is a change in the remaining record, the record should get indicator ‘C’.
  Sample data looks like this.</p>
  
  <p>Old File:</p>
  
  <pre><code>Key Value
  10000   Apple
  20000   Orange
  30000   Melon
  40000   Berry
  50000   Cherry
  </code></pre>
  
  <p>New File:</p>
  
  <pre><code>Key Value
  10001   Pear
  20002   Pineapple
  30000   Melon
  40000   Strawberry
  50000   SweetCherry
  </code></pre>
  
  <p>Expected Result:</p>
  
  <pre><code>Key Value   Indicator
  10000   Apple   D
  20000   Orange  D
  10001   Pear    A
  20002   Pineapple   A
  40000   Strawberry  C
  50000   SweetCherry C
  </code></pre>
  
  <p>I have written the following program to achieve this.</p>
  
  <pre><code>import org.apache.spark.{SparkConf, SparkContext}
  object transaction {
    def main(args: Array[String]): Unit = {
      val conf = new SparkConf().setMaster("local").setAppName("transaction")
      val sc = new SparkContext(conf)
      val oldFile = sc.textFile("hdfs://quickstart.cloudera:8020/user/root/OLDFILE.txt")
      val newFile = sc.textFile("hdfs://quickstart.cloudera:8020/user/root/NEWFILE.txt")
      val oldMap = oldFile.map(rec =&gt; (rec.substring(0,9), rec))
      val newMap = newFile.map(rec =&gt; (rec.substring(0,9), rec))
      val tranJoin = oldMap.fullOuterJoin(newMap)
      val tranFinal = tranJoin.map(rec =&gt; {
        if (rec._2._1 == None)
          rec._2._2.mkString+" A"
        else
        if (rec._2._2 == None)
          rec._2._1.mkString+" D"
        else
        if (rec._2._1 != rec._2._2)
          rec._2._2.mkString+" C"
      }).filter(rec =&gt; rec.toString.substring(0,1) != "(")
   tranFinal.saveAsTextFile("hdfs://quickstart.cloudera:8020/user/root/transfinal")
  }
  </code></pre>
  
  <p>I would like to know if there is a more functional way to write this program.</p>