title: Spark Streaming: HDFS
body: <ol>
  <li>I can't get my Spark job to stream "old" files from HDFS. </li>
  </ol>
  
  <p>If my Spark job is down for some reason (e.g. demo, deployment) but the writing/moving to HDFS directory is continuous, I might skip those files once I up the Spark Streaming Job.</p>
  
  <pre><code>    val hdfsDStream = ssc.textFileStream("hdfs://sandbox.hortonworks.com/user/root/logs")
  
      hdfsDStream.foreachRDD(
        rdd =&gt; logInfo("Number of records in this batch: " + rdd.count())
      )
  </code></pre>
  
  <p>Output --> Number of records in this batch: 0</p>
  
  <ol start="2">
  <li><p>Is there a way for Spark Streaming to move the "read" files to a different folder? Or we have to program it manually? So it will avoid reading already "read" files.</p></li>
  <li><p>Is Spark Streaming the same as running the spark job (sc.textFile) in CRON? </p></li>
  </ol>