title: Chaining spark sql queries over temporary views?
body: <p>We're exploring the possibility to use temporary views in spark and relate this to some actual file storage - or to other temporary views. We want to achieve something like:</p>
  
  <ol>
  <li>Some user uploads data to some S3/hdfs file storage.</li>
  <li>A (temporary) view is defined so that spark sql queries can be run against the data.</li>
  <li>Some other temporary view (referring to some other data) is created.</li>
  <li>A third temporary view is created that joins data from (2) and (3).</li>
  </ol>
  
  <p>By selecting from the view from (4) the user gets a table that reflects the joined data from (2) and (3). This result can be further processed and stored in a new temporary view and so on.</p>
  
  <p>So we end up with a tree of temporary views - querying their parent temporary views until they end up loading data from the filesystem. Basically we want to store transformation steps (selecting, joining, filtering, modifying etc) on the data - without storing new versions. The spark SQL-support and temporary views seems like a good fit.</p>
  
  <p>We did some successful testing. The idea is to store the specification of these temporary views in our application and recreate them during startup (as temporary or global views).</p>
  
  <p>Not sure if this is viable solution? One problem is that we need to know how the temporary views are related (which one queries which). We create them like:</p>
  
  <p><code>sparkSession.sql("select * from other_temp_view").createTempView(name)</code></p>
  
  <p>So, when this is run we have to make sure that <code>other_temp_view</code> is already created in the session. Not sure how this can be achieved. One idea is to store a timestamp and recreate them in the same order. This could be ok since out views most likely will have to be "immutable". We're not allowed to change a query that other queries relies on.</p>
  
  <p>Any thoughts would be most appreciated.</p>