title: Why is pyspark implemented such that exiting a session is stopping the underlying spark context?
body: <p>I just massively shot my foot by writing &quot;pythonic&quot; spark code like this:</p>
  <pre class="lang-py prettyprint-override"><code># spark = ... getOrCreate() # essentially provided by the environment (Databricks)
  with spark.newSession() as session:
       session.catalog.setCurrentDatabase(&quot;foo_test&quot;)
       do_something_within_database_scope(session)
  
  assert spark.currentDatabase() == &quot;default&quot;
  </code></pre>
  <p>And oh was I surprised that when executing this notebook cell, somehow the cluster terminated.</p>
  <p>I read through <a href="https://stackoverflow.com/a/41513805/2160256">this answer</a> which tells me, that there can only be one spark context. That is fine. But why is exiting a session terminating the underlying context? Is there some requirement for this or is this just a design flaw in pyspark?</p>
  <p>I also understand that the session's <code>__exit__</code> call invokes <code>context.stop()</code> - I want to know why it is implemented like that!</p>
  <p>I always think of a <code>session</code> as some user initiated thing, like with databases or http clients which I can create and discard on my own will. If the session provides <code>__enter__</code> and <code>__exit__</code> then I try to use it from within a <code>with</code> context to make sure I clean up after I am done.</p>
  <p>Is my understanding wrong, or alternatively why does pyspark deviate from that concept?</p>
  <p><em>Edit: I tested this together with <code>databricks-connect</code> which comes with its own pyspark python module, but as pri pointed out below it seems to be implemented the same way in standard pyspark.</em></p>