title: Understanding Spark's caching
body: <p>I'm trying to understand how Spark's cache work. </p>
  
  <p>Here is my naive understanding, please let me know if I'm missing something:</p>
  
  <pre><code>val rdd1 = sc.textFile("some data")
  rdd1.cache() //marks rdd1 as cached
  val rdd2 = rdd1.filter(...)
  val rdd3 = rdd1.map(...)
  rdd2.saveAsTextFile("...")
  rdd3.saveAsTextFile("...")
  </code></pre>
  
  <p>In the above, rdd1 will be loaded from disk (e.g. HDFS) only once. (when rdd2 is saved I assume) and then from cache (assuming there is enough RAM) when rdd3 is saved)</p>
  
  <p>Now here is my question. Let's say I want to cache rdd2 and rdd3 as they will both be used later on, but I don't need rdd1 after creating them. </p>
  
  <p>Basically there is duplication, isn't it? Since once rdd2 and rdd3 are calculated, I don't need rdd1 anymore, I should probably unpersist it, right? the question is when? </p>
  
  <p><strong>Will this work? (Option A)</strong></p>
  
  <pre><code>val rdd1 = sc.textFile("some data")
  rdd1.cache()   // marks rdd as cached
  val rdd2 = rdd1.filter(...)
  val rdd3 = rdd1.map(...)
  rdd2.cache()
  rdd3.cache()
  rdd1.unpersist()
  </code></pre>
  
  <p>Does spark add the unpersist call to the DAG? or is it done immediately? if it's done immediately, then basically rdd1 will be non cached when I read from rdd2 and rdd3, right? </p>
  
  <p><strong>Should I do it this way instead (Option B)?</strong></p>
  
  <pre><code>val rdd1 = sc.textFile("some data")
  rdd1.cache()   // marks rdd as cached
  val rdd2 = rdd1.filter(...)
  val rdd3 = rdd1.map(...)
  
  rdd2.cache()
  rdd3.cache()
  
  rdd2.saveAsTextFile("...")
  rdd3.saveAsTextFile("...")
  
  rdd1.unpersist()
  </code></pre>
  
  <p><strong>So the question is this:</strong>
  Is Option A good enough? i.e. will <code>rdd1</code> still load the file only once?
  Or do I need to go with Option B?</p>