title: Exception handling in the Spark streaming application
body: <p>I have developed a Spark Streaming application which checkpoints a filestream. I need to stop my streaming application on any drive exception..my codes are as follow:</p>
  
  <pre><code>val fileStream=..
    fileStream.checkpoint(Duration(batchIntervalSeconds * 1000 * 5))
  
  //initiate the chekpointing
  fileStream.foreachRDD(r=&gt; {
    try {
      r.count()
    } catch {
      case ex: Exception =&gt; {
        ssc.stop(true, true)
      }
  
    }
  }
  )
  </code></pre>
  
  <p>However, I get the exception from the above code</p>
  
  <pre><code> yarn.ApplicationMaster: User class threw exception:
  java.io.NotSerializableException: DStream checkpointing has been enabled but the DStreams with their functions are not serializable
  org.apache.spark.streaming.StreamingContext
  Serialization stack:
      - object not serializable (class: org.apache.spark.streaming.StreamingContext, value: org.apache.spark.streaming.StreamingContext@45ae9d8b)
      - field (class: UnionStream$$anonfun$creatingFunc$3, name: ssc$1, type: class org.apache.spark.streaming.StreamingContext)
      - object (class UnionStream$$anonfun$creatingFunc$3, &lt;function1&gt;)
      - field (class: org.apache.spark.streaming.dstream.DStream$$anonfun$foreachRDD$1$$anonfun$apply$mcV$sp$3, name: cleanedF$1, type: interface scala.Function1)
      - object (class org.apache.spark.streaming.dstream.DStream$$anonfun$foreachRDD$1$$anonfun$apply$mcV$sp$3, &lt;function2&gt;)
      - writeObject data (class: org.apache.spark.streaming.dstream.DStream)
      - object (class org.apache.spark.streaming.dstream.ForEachDStream, org.apache.spark.streaming.dstream.ForEachDStream@12481647)
      - writeObject data (class: org.apache.spark.streaming.dstream.DStreamCheckpointData)
      - object (class org.apache.spark.streaming.dstream.DStreamCheckpointData, 
  </code></pre>
  
  <p>I am running my codes on yarn cluster mode..</p>