title: Efficient way to join a cached spark dataframe with other and cache again
body: <p>I have a large dataframe that has been cached like</p>
  
  <pre><code>val largeDf = someLargeDataframe.cache
  </code></pre>
  
  <p>Now I need to union it with a tiny one and cached it again</p>
  
  <pre><code>val tinyDf = someTinyDataframe.cache
  val newDataframe = largeDf.union(tinyDf).cached
  tinyDf.unpersist()
  largeDf.unpersist()
  </code></pre>
  
  <p>It is very inefficient since it need to re-cached all the data again. Is there any efficient way to add a little amount of data to a large cached dataframe? </p>
  
  <hr>
  
  <p>After reading Teodors's explanation, I know that I can't unpersist the old dataframe before I do some action on my new dataframe. But what if I need to do something like this?</p>
  
  <pre><code>def myProcess(df1: Dataframe, df2: Dataframe): Dataframe{
      val df1_trans = df1.map(....).cache
      val df2_trans = df2.map(....).cache
  
      doSomeAction(df1_trans, df2_trans)
  
      val finalDf = df1_trans.union(df2_trans).map(....).cache
      // df1_trans.unpersist()
      // df2_trans.unpersist()
      finalDf
  }
  </code></pre>
  
  <p>I want my df1_trans &amp; df2_trans to be cached to improve the performance inside the function since they will be called more than once, but the dataframe I need to return in the end is also constructed by df1_trans &amp; df2_trans, if I can't unpersist them before leaving the function, I can never find other place to do this, however, if I unpersist them, my finalDf will not benefit from cache.</p>
  
  <p>What can I do in this situation? Thanks!</p>