title: Spark streaming checkpoints for DStreams
body: <p>In Spark Streaming it is possible (and mandatory if you're going to use stateful operations) to set the <code>StreamingContext</code> to perform checkpoints into a reliable data storage (S3, HDFS, ...) of (AND):</p>
  
  <ul>
  <li>Metadata</li>
  <li><code>DStream</code> lineage</li>
  </ul>
  
  <p>As described <a href="http://spark.apache.org/docs/latest/streaming-programming-guide.html#how-to-configure-checkpointing">here</a>, to set the output data storage you need to call <code>yourSparkStreamingCtx.checkpoint(datastoreURL)</code></p>
  
  <p>On the other hand, it is possible to set lineage checkpoint intervals for each <code>DataStream</code> by just calling <code>checkpoint(timeInterval)</code> at them. In fact, it is recommended to set lineage checkpoint interval between 5 and 10 times the <code>DataStream</code>'s sliding interval:</p>
  
  <blockquote>
    <p>dstream.checkpoint(checkpointInterval). Typically, a checkpoint
    interval of 5 - 10 sliding intervals of a DStream is a good setting to
    try.</p>
  </blockquote>
  
  <p>My question is:</p>
  
  <p>When the streaming context has been set to perform checkpointing and <strong>no <code>ds.checkpoint(interval)</code> is called</strong>, is lineage checkpointing enabled for all data streams with a default <code>checkpointInterval</code> equal to <code>batchInterval</code>? Or is, on the contrary, only metadata checkpointing what is enabled?</p>