title: What is the purpose of cache an RDD in Apache Spark?
body: <p>I am new for Apache Spark and I have couple of basic questions in spark which I could not understand while reading the spark material. Every materials have their own style of explanation. I am using PySpark Jupyter notebook on Ubuntu to practice. </p>
  
  <p>As per my understanding, When I run the below command, the data in the testfile.csv is partitioned and stored in memory of the respective nodes.( actually I know its a lazy evaluation and it will not process until it sees action command ), but still the concept is</p>
  
  <pre><code>rdd1 = sc.textFile("testfile.csv")
  </code></pre>
  
  <p>My question is when I run the below transformation and action command, where does the rdd2 data will store. </p>
  
  <p>1.Does it stores in memory?</p>
  
  <pre><code>rdd2 = rdd1.map( lambda x: x.split(",") )
  
  rdd2.count()
  </code></pre>
  
  <p>I know the data in rdd2 will available till I close the jupyter notebook.Then what is the need of cache(), anyhow rdd2 is available to do all transformation. I heard after all the transformation the data in memory is cleared, what is that about?</p>
  
  <ol start="2">
  <li><p>Is there any difference between keeping RDD in memory and cache()</p>
  
  <p>rdd2.cache()</p></li>
  </ol>