title: How to configure Spark spark_worker_opts for Jupyter notebooks
body: <p>I use Pyspark with Spark 2.4 in the standalone mode on Linux for processing a lot of incoming data via Kafka using a Jupyter notebook (currently for testing). I want to add these options to this notebook in order to prevent the /tmp/ directory to be filled with dozens of gigabytes after few hours:</p>
  <pre><code>spark.worker.cleanup.enabled=true
  spark.worker.cleanup.appDataTtl=120
  </code></pre>
  <p>But these conf locations do not work:</p>
  <ol>
  <li>Spark’s default configuration (spark/conf/spark-env.sh) seems not be used by Juypter notebooks at all:</li>
  </ol>
  <blockquote>
  <p>SPARK_WORKER_OPTS=&quot;spark.worker.cleanup.enabled=true
  spark.worker.cleanup.appDataTtl=120&quot;</p>
  </blockquote>
  <ol start="2">
  <li>So, I created a sperate kernel configuration in ~/.local/share/jupyter/kernels/python3-spark1/kernel.json that I can select in Jupyterhub and that is really used for the RAM adjustments what I can see in htop:</li>
  </ol>
  <blockquote>
  <p>&quot;env&quot;: {
  &quot;PYSPARK_SUBMIT_ARGS&quot;: &quot;--master local[*]
  --conf spark.worker.cleanup.enabled=true --conf=spark.worker.cleanup.appDataTtl=120 driver-memory 145g --executor-memory 50g pyspark-shell&quot;</p>
  </blockquote>
  <p>but the /tmp still fills with dozens of gigs.</p>
  <ol start="3">
  <li>I also tried the “magic” in a jupyter cell but it also did not work.</li>
  </ol>
  <p>Do you know how to configure the Jupyter notebooks for this Spark adjustments properly?</p>