title: How is spark a memory based solution if it writes data to disk before shuffles?
body: <p>One of the main &quot;selling points&quot; of Spark is that unlike Hadoop map-reduce that persists intermediate results of its computation to hdfs, Spark keeps all its results in memory. I don't understand this as in reality when a Spark stage finishes <a href="https://github.com/JerryLead/SparkInternals/blob/master/markdown/english/4-shuffleDetails.md" rel="nofollow noreferrer">it writes all of the data into shuffle files stored on the disk</a>. How then is this an improvement on map-reduce?</p>
  <p><a href="https://i.stack.imgur.com/bezYN.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/bezYN.png" alt="Image taken from Advanced Apache Spark Training - Sameer Farooqui" /></a></p>